Controller and control method

ABSTRACT

The present technology relates to a controller and a control method capable of providing an imaging system which achieves cost reduction. 
     A control unit controls a first imaging direction of a first imaging unit and a second imaging direction of a second imaging unit as a direction different from the first imaging direction in accordance with a situation of an object associated with a first image captured by the first imaging unit or a second image captured by the second imaging unit. For example, the present technology is applicable to a lecture capture system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/771,870, filed on Jun. 11, 2020, and is based upon and claims thebenefit of priority to International Application No. PCT/JP2018/045032,filed on Dec. 7, 2018 and JP 2017-244732, filed on Dec. 21, 2017. Theentire contents of U.S. Ser. No. 16/771,870 are incorporated herein byreference.

TECHNICAL FIELD

The present technology relates to a controller and a control method, andparticularly to a controller and a control method capable of providingan imaging system which achieves cost reduction.

BACKGROUND ART

A lecture capture system has been provided in recent years to record astate of a lecture given in a college or other schools, and allowattendance at the lecture in a remote place.

PTL 1 discloses a lecture capture system which images a lecturer and anauditor to track the lecturer and detect a standing-up action of theauditor, and switches between videos of the lecturer and the auditor ormerges the videos into one video for outputting, as necessary.

For tracking the lecturer and detecting the standing-up action of theauditor, the system of PTL 1 includes two overhead cameras capable ofcapturing overhead views of the lecturer and the auditor, and twotracking cameras tracking the lecturer and the auditor detected by therespective overhead cameras. Each of the tracking cameras is adrive-type camera capable of performing panning, tilting, and zooming,and can capture and image an object at an appropriate angle of view.

CITATION LIST Patent Literature [PTL 1]

JP 2005-33570 A

SUMMARY Technical Problem

Meanwhile, if imaging in different directions is conventionallycontrollable in accordance with a situation of an object correspondingto an imaging target of each of the directions, reduction of the numberof cameras, and therefore cost reduction of the system are achievable.

The present technology has been developed in consideration of theaforementioned circumstances, and provides an imaging system whichachieves cost reduction.

Solution to Problem

A controller of the present technology includes a control unit thatcontrols a first imaging direction of a first imaging unit and a secondimaging direction of a second imaging unit as a direction different fromthe first imaging direction in accordance with a situation of an objectassociated with a first image captured by the first imaging unit or asecond image captured by the second imaging unit.

A control method according to the present technology is a control methodof a controller. The control method includes controlling a first imagingdirection of a first imaging unit and a second imaging direction of asecond imaging unit as a direction different from the first imagingdirection in accordance with a situation of an object associated with afirst image captured by the first imaging unit or a second imagecaptured by the second imaging unit.

According to the present technology, the first imaging direction of thefirst imaging unit and the second imaging direction of the secondimaging unit, which direction is different from the first imagingdirection, are controlled in accordance with the situation of the objectassociated with the first image captured by the first imaging unit orthe second image captured by the second imaging unit.

Advantageous Effects of Invention

According to the present technology, an imaging system which achievescost reduction can be provided.

Note that advantageous effects to be offered are not necessarily limitedto the advantageous effects described herein, but may be anyadvantageous effects described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting a configuration example of an imagingsystem to which the present technology is applied.

FIG. 2 is a block diagram depicting a functional configuration exampleof a camera controller.

FIG. 3 is a diagram depicting an example of a preset position set by acamera control unit.

FIG. 4 is a block diagram depicting a functional configuration exampleof a controller.

FIG. 5 is a diagram depicting a description example of a scenario.

FIG. 6 is a diagram depicting details of States.

FIG. 7 is a diagram depicting details of States.

FIG. 8 is a diagram depicting details of States.

FIG. 9 is a diagram depicting an action example of the imaging systembased on the scenario.

FIG. 10 is a diagram depicting an action example of the imaging systembased on the scenario.

FIG. 11 is a diagram depicting an action example of the imaging systembased on the scenario.

FIG. 12 is a diagram depicting an action example of the imaging systembased on the edited scenario.

FIG. 13 is a diagram depicting a description example of the scenarioafter editing.

FIG. 14 is a diagram depicting details of States.

FIG. 15 is a diagram depicting an example of GUI used for editing of thescenario.

FIG. 16 is a diagram depicting an example of GUI used for editing of thescenario.

FIG. 17 is a diagram depicting an example of GUI used for editing of thescenario.

FIG. 18 is a diagram depicting an example of GUI used for defining andregistering a gesture.

FIG. 19 is a diagram depicting another configuration example of theimaging system.

FIG. 20 is a block diagram depicting a functional configuration exampleof a camera controller.

FIG. 21 is a diagram depicting a description example of a scenario.

FIG. 22 is a diagram depicting details of States.

FIG. 23 is a diagram depicting details of States.

FIG. 24 is a diagram depicting an action example of the imaging systembased on the scenario.

FIG. 25 is a diagram depicting a further configuration example of theimaging system.

FIG. 26 is a diagram depicting a description example of a scenario.

FIG. 27 is a diagram depicting details of States.

FIG. 28 is a diagram depicting details of States.

FIG. 29 is a diagram depicting details of States.

FIG. 30 is a diagram depicting an action example of the imaging systembased on the scenario.

FIG. 31 is a diagram depicting a still further configuration example ofthe imaging system.

FIG. 32 is a block diagram depicting a functional configuration exampleof a camera controller.

FIG. 33 is a diagram depicting details of States.

FIG. 34 is a diagram depicting a description example of a scenarioexpressed by a timeline.

FIG. 35 is a block diagram depicting a configuration example of acomputer.

FIG. 36 is a view schematically depicting a general configuration of asurgery room system.

FIG. 37 is a view depicting an example of display of an operation screenimage of a centralized operation panel.

FIG. 38 is a view illustrating an example of a state of surgery to whichthe surgery room system is applied.

FIG. 39 is a block diagram depicting an example of a functionalconfiguration of a camera head and a camera control unit (CCU) depictedin FIG. 38.

DESCRIPTION OF EMBODIMENTS

Modes for carrying out the present disclosure (hereinafter referred toas embodiments) will be hereinafter described. Note that the descriptionwill be presented in a following order.

0. Problems arising from conventional lecture capture system

1. First Embodiment (Basic Configuration and Action of Imaging System)

2. Second Embodiment (Configuration Outputting Contents Written OnBoard)

3. Third Embodiment (Configuration Outputting Presentation Slide)

4. Other variation examples

5. Application examples

0. PROBLEMS ARISING FROM CONVENTIONAL LECTURE CAPTURE SYSTEM

There is conventionally known such a lecture capture system which imagesa lecturer and an auditor to track the lecturer and detect a standing-upaction of the auditor, and switches between videos of the lecturer andthe auditor or merges the videos into one video for outputting, asnecessary.

For tracking the lecturer and detecting the standing-up action of theauditor, this system includes two overhead cameras capable of capturingoverhead views of the lecturer and the auditor, and two tracking camerastracking the lecturer and the auditor detected by the respectiveoverhead cameras.

However, following problems are arising from the conventional lecturecapture system.

(Problem 1)

This system requires overhead cameras each detecting an object inaddition to tracking cameras each tracking and imaging the object.Accordingly, costs of the system increase.

(Problem 2)

Moreover, action procedures of the entire system described above (e.g.,zooming-in the auditor by the tracking cameras at the time of detectionof the standing-up action of the auditor by the overhead cameras) aredetermined by a system designer beforehand. Accordingly, any change ofthe actions of the system is generally difficult to make by a user.

For example, while a lecture style varies for each lecturer, thelecturer of the conventional system is required to be conscious of theactions determined beforehand while giving a lecture. In this case, thelecture cannot be given in an original style.

In addition, according to the conventional system, changes of the systemconfiguration (e.g., increase or decrease of the number of trackingcameras, and use of a combination of cameras having different functionsand performance) are also difficult to flexibly handle.

(Problem 3)

Furthermore, the conventional system is designed chiefly for the purposeof imaging a lecturer and an auditor. In this case, contents written ona board during a lecture and slides for presentation are not clearlyrecorded. Accordingly, detailed contents on the board and thepresentation are difficult to recognize by an audience.

In consideration of the above, hereinafter described will beconfigurations and actions of a system capable of solving theaforementioned problems.

According to an imaging system of the present embodiments, a controlunit controls a first imaging direction of a first camera, and a secondimaging direction of a second camera as a direction different from thefirst imaging direction in accordance with a situation of an objectassociated with a first image or a second image. A range of imaging bythe first camera is controlled by controlling the first imagingdirection, while a range of imaging by the second camera is controlledby controlling the second imaging direction.

The object associated with the first image or the second image mayinclude not only a person such as a lecturer and an auditor describedbelow, but also a blackboard or a whiteboard used for board writing, andfurther slides and materials outputted from a PC (Personal Computer).

1. FIRST EMBODIMENT (Configuration Example of Imaging System)

FIG. 1 is a diagram depicting a configuration example of an imagingsystem to which the present technology is applied.

An imaging system depicted in FIG. 1 is configured as a lecture capturesystem, and provided in a classroom, a lecture hall or the like where alecturer L1 gives a lecture to a plurality of auditors A1.

The imaging system in FIG. 1 includes a lecturer camera 11-1, an auditorcamera 11-2, a lecturer camera controller 12-1, an auditor cameracontroller 12-2, and a controller 13. An input/output apparatus 14 and arecording apparatus 15 are added to the controller 13, and aninformation processing terminal 17 is further connected to thecontroller 13 via a network 16.

The lecturer camera 11-1 as the first imaging unit captures an image ofthe lecturer L1, and outputs the captured image (lecturer camera image)to the lecturer camera controller 12-1 under control by the lecturercamera controller 12-1.

The auditor camera 11-2 as the second imaging unit captures an image ofthe auditors A1 located in a direction different from a direction wherethe lecturer L1 is located, and outputs the captured image (auditorcamera image) to the auditor camera controller 12-2 under control by theauditor camera controller 12-2.

For example, the lecturer camera 11-1 and the auditor camera 11-2 aredisposed in such positions as to face each other (such positions thatrespective imaging directions face each other) in a front part (platformside) and a rear part of the classroom, respectively, to image in thedirection where the lecturer L1 is located, and in the direction wherethe auditors A1 are located.

The lecturer camera 11-1 and the auditor camera 11-2 may have a similarconfiguration except for only a difference of an object corresponding toan imaging target, i.e., whether the object is the lecturer L1 or theauditors A1. For example, the lecturer camera 11-1 and the auditorcamera 11-2 are each configured as a PTZ camera which has a function ofmechanical panning and tilting, and also are each capable of performingoptical and electronical zooming.

Note that the lecturer 11-1 and the auditor camera 11-2 are each onlyrequired to image objects located in different directions (in otherwords, different types of object) without the necessity of equipping anoverhead camera. The positions of the respective cameras 11-1 and 11-2are not limited to the positions described above.

For example, assuming that the front side of the classroom is located inthe direction of 12 o′clock, the lecturer camera 11-1 is disposed in thevicinity of the center of the classroom in such a position as to imagein the direction of 12 o′clock. In this case, it is sufficient that theauditor camera 11-2 is disposed in the vicinity of the center of theclassroom in such a position as to image in the direction of 6 o′clock,or image in the direction of 3 o′clock or 9 o′clock in correspondencewith the direction where the auditors A1 are located.

In addition, in a case where the two lecturers L1 are present, forexample, assuming that the front side of the classroom is located in thedirection of 12 o′clock, the two lecturer cameras 11-1 are disposed inthe vicinity of the center of the classroom in such positions as toimage in the direction of 10 o′clock and 2 o′clock, respectively.

Furthermore, the lecturer camera 11-1 and the auditor camera 11-2 may beconfigured as a single 360-degree camera or a fish-eye camera (oneimaging apparatus). In this case, the lecturer camera image and theauditor camera image are obtained by cutting an image in the directionwhere the lecturer L1 is located, and an image in the direction wherethe auditors A1 are located from an omnidirectional image and awide-angle image obtained by the 360-degree camera or the fish-eyecamera.

As described above, the imaging system to which the present technologyis applied is not required to have the configuration which uses aplurality of cameras so disposed as to image in different directions,but may have other configurations as long as images having differentimaging ranges (angles of view and imaging directions) can be obtained.

The lecturer camera 11-1 and the auditor camera 11-2 are hereinaftersimply referred to as cameras 11 in a case where no distinction is madebetween the lecturer camera 11-1 and the auditor camera 11-2. Imagesoutputted from the cameras 11 are referred to as camera images. Inaddition, it is assumed hereinafter that the camera images are movingimages. However, the camera images may be still images, or continuousimages obtained by continuously imaging in terms of time.

The lecturer camera controller 12-1 controls the angle of view or theimaging direction of the lecturer camera 11-1 under control by thecontroller 13 to control the imaging range of the lecturer camera 11-1.The lecturer camera image outputted from the lecturer camera 11-1 isoutputted to the controller 13. In addition, the lecturer cameracontroller 12-1 receives an instruction of calculation and control fromthe controller 13, and outputs a result of calculation and control tothe controller 13.

The auditor camera controller 12-2 controls the angle of view or theimaging direction of the auditor camera 11-2 under control by thecontroller 13 to control the imaging range of the auditor camera 11-2.The auditor camera image outputted from the auditor camera 11-2 isoutputted to the controller 13. In addition, the auditor cameracontroller 12-2 receives an instruction of calculation and control fromthe controller 13, and outputs a result of calculation and control tothe controller 13.

The lecturer camera controller 12-1 and the auditor camera controller12-2 may have a similar configuration except for only a difference in acontrol target, i.e., whether the control target is the lecturer camera11-1 or the auditor camera 11-2.

The lecturer camera controller 12-1 and the auditor camera controller12-2 are hereinafter simply referred to as camera controllers 12 in acase where no distinction is made between the lecturer camera controller12-1 and the auditor camera controller 12-2. Note that the cameracontrollers 12 may be built in the cameras 11.

In addition, in a case where the cameras 11 are configured as a single360-degree camera or fish-eye camera, the single camera controller 12 isprovided to control ranges of the lecturer camera image and the auditorcamera image cut from an omnidirectional image or a wide-angle image.

As described above, the camera controllers 12 control imaging ranges ofthe lecturer camera image and the auditor camera image by controllingangles of view and imaging directions of the cameras 11, or ranges cutfrom an omnidirectional image or a wide-angle image.

The controller 13 controls the plurality of camera controllers 12(lecturer camera controller 12-1 and auditor camera controller 12-2) toacquire camera images outputted from the respective camera controllers12, and output the camera images switched or merged into one image, forexample. The outputted camera images are recorded in the recordingapparatus 15, or transmitted to the information processing terminal 17via the network 16.

In addition, the controller 13 outputs an instruction of calculation andcontrol to each of the camera controllers 12, and acquires a result ofcalculation and control from each of the camera controllers 12. Thecontroller 13 controls the camera controllers 12 on the basis of theresults of calculation and control received from the camera controllers12.

Note that the camera controllers 12 and the controller 13 may eachinclude dedicated hardware having corresponding functions, or include anordinary computer and implement corresponding functions by software. Inaddition, the camera controllers 12 and the controller 13 may beconfigured not as components separated from each other, but ascomponents provided integrally with each other as one controller.

The input/output apparatus 14 includes a keyboard and a mouse receivingan operation from a user, and also a display and the like having adisplay function. This display may have a function of a touch panel. Thecontroller 13 defines a scenario which represents action procedures ofrespective apparatuses constituting the imaging system (how to implementactions of respective apparatuses) on the basis of operations performedby the system designer or the user and received via the input/outputapparatus 14 to control the actions of the imaging system. Details ofthe scenario will be described below.

The information processing terminal 17 is a terminal used by an audiencelocated within a space where a lecture is given by the lecturer L1 or ina remote place to view and listen to images outputted from thecontroller 13.

The information processing terminal 17 may include dedicated hardwarehaving a corresponding function, or may include an ordinary computer andimplement the corresponding function by software. In addition, a camera,a microphone or the like may be provided on the information processingterminal 17 to transmit images and voices from the space where theaudience is located to the controller 13 via the network 16.

(Functional Configuration Example of Camera Controller)

FIG. 2 is a block diagram depicting a functional configuration exampleof the camera controllers 12 (lecturer camera controller 12-1 and theauditor camera controller 12-2) described above.

The camera controller 12 includes a moving body detection unit 21, agesture detection unit 22, a tracking unit 23, and a camera control unit24.

The moving body detection unit 21 acquires a camera image (image signal)from each of the cameras 11, and detects a moving body in the cameraimage under control by the camera control unit 24. Informationindicating a position (e.g., coordinates) and a shape of the detectedmoving body is supplied to the camera control unit 24.

The gesture detection unit 22 acquires a camera image (image signal)from each of the cameras 11, and detects a gesture (action) of an object(lecturer L1 or auditor A1) in the camera image under control by thecamera control unit 24. Information indicating a type and a posture ofthe detected gesture is supplied to the camera control unit 24.

The tracking unit 23 acquires a camera image (image signal) from each ofthe cameras 11, and tracks an object in the camera image under controlby the camera control unit 24. Information indicating a position (e.g.,coordinates) and a shape of an object corresponding to a tracking targetis supplied to the camera control unit 24.

The camera control unit 24 controls the respective units provided as themoving body detection unit 21, the gesture detection unit 22, and thetracking unit 23, and calculates appropriate angles of view andcamerawork of the cameras 11 on the basis of information obtained fromthe respective units as a result of the control. The camera control unit24 controls the cameras 11 on the basis of a camera control signalindicating a result of this calculation.

Moreover, the camera control unit 24 exchanges information provided forexecuting the scenario described below, information indicating states ofthe cameras 11 and results of calculation and control, and the like withthe controller 13 as apparatus control signals.

When the camera control unit 24 controls the respective units providedas the moving body detection unit 21, the gesture detection unit 22, andthe tracking unit 23, a preset position indicated by a frame PF in FIG.3 is set as a preset position of the angle of view of the lecturercamera image, for example. In FIG. 3, the preset position (frame PF) isset to a range around a blackboard and a platform (teacher's desk) ofthe classroom.

When the lecturer L1 stands on the platform, for example, i.e., when amoving body is detected within the frame PF preset in the lecturercamera image by the moving body detection unit 21, the camera controlunit 24 shifts the control target to the tracking unit 23 in response tothis detection as a trigger. The camera control unit 24 calculates anappropriate angle of view of the lecturer camera 11-1 on the basis ofinformation obtained as a result of tracking of the object (lecturer L1)by the tracking unit 23, and generates a camera control signal forcontrolling panning, tilting, and zooming to control the lecturer camera11-1.

(Functional Configuration Example of Controller)

FIG. 4 is a block diagram depicting a functional configuration exampleof the controller 13 described above.

The controller 13 includes a center control unit 31, a scenariogeneration unit 32, and a switcher 33.

The center control unit 31 executes action control of the entire imagingsystem on the basis of the scenario (described below) definedbeforehand. For example, the center control unit 31 integratedlycontrols imaging of the lecturer camera 11-1 (in the direction where thelecturer L1 is located), and imaging of the auditor camera 11-2 (in thedirection where the auditors A1 are located) in accordance with asituation of an object associated with the lecturer camera image or theauditor camera image.

More specifically, the center control unit 31 acquires the lecturercamera image by controlling the lecturer camera controller 12-1, andacquires the auditor camera image by controlling the auditor cameracontroller 12-2. Thereafter, the center control unit 31 controls imagingby the respective cameras 11 (actions of the respective cameracontrollers 12) in accordance with the situation of the object in thelecturer camera image or the auditor camera image under the scenario,and controls output of at least any one of the obtained camera images(action of the switcher 33).

For example, the situation of the object refers to a behavior of theobject. The behavior of the object may be either an action of theobject, or a sound emitted from the object. Specific examples of theaction of the object include standing up, sitting down, raising hand,applause, respective actions such as body languages and pointing carriedout during explanation of matters, movement to the preset positiondescribed above, movement from the preset position, and others detectedby gesture detection. Examples of the sound emitted from the objectinclude spoken voices such as a call given to another person and a replyto this call, sounds of applause, chimes, and others.

More specifically, according to the imaging system of the presentembodiment, the angles of view and the imaging directions of the cameras11 (the lecturer camera 11-1 and the auditor camera 11-2) (respectiveimaging ranges of the cameras 11) are controlled in accordance with thesituation of the object as described above.

In addition to this control, the imaging directions of the cameras 11may be controlled in a case where an overlap between a plurality ofobjects is detected in the camera images acquired by the cameras 11 asthe situation of the object, for example. Moreover, the angles of viewand the imaging directions of the cameras 11 may be controlled in a casewhere the same type of object is detected in the camera images acquiredby the cameras 11 as the situation of the object, for example.

The scenario is information which describes shifts of states of therespective camera controllers 12 and the switcher 33 corresponding toaction control targets. More specifically, the scenario describesactions (Behaviors) executed by the respective control targets, andtriggers (Events) for shifting to other states for each of a pluralityof states (States) of the entire imaging system.

The center control unit 31 issues apparatus control signals to therespective controllers 12 and the switcher 33 to request the respectivecontrollers 12 and the switcher 33 to perform Behaviors and Eventsdescribed in the scenario. In this manner, the center control unit 31manages the states (States) of the respective camera controllers 12 andthe switcher 33, and controls the actions of the respective cameracontrollers 12 and the switcher 33. A basic dialog with the user, suchas a start and an end of execution of the scenario, and selection of thescenario to be executed, is achieved on the basis of an operation signalsupplied from the input/output apparatus 14.

The center control unit 31 may execute not only the scenario definedbeforehand, but also a scenario generated by the scenario generationunit 32.

The scenario generation unit 32 enables the user to define and generatea scenario. A basic dialog with the user for generating a scenario isachieved on the basis of an operation signal supplied from theinput/output apparatus 14. The generated scenario is supplied to thecenter control unit 31.

The switcher 33 as an output control unit acquires camera images (imagesignals) outputted from the respective camera controllers 12, andoutputs any one of the image signals after switching to this imagesignal, or outputs an image signal produced by merging a plurality ofthe image signals under control by the center control unit 31. Theoutputted image signal is recorded in the recording apparatus 15, ortransmitted to the information processing terminal 17 via the network16.

Examples of merging of camera images (image signals) by the switcher 33include side-by-side merging in which two images are arranged side byside, picture-in-picture merging in which one image is displayed insideanother image, overlaying in which two images are overlaid, and others.

While the switcher 33 in the example of FIG. 4 is built in thecontroller 13, note that the switcher 33 may be configured as anindependent apparatus separated from the controller 13.

(Description Example of Scenario)

A description example of the scenario will be herein described withreference to FIG. 5.

As described in FIG. 5, the scenario includes at least one Entry Point(EP), and at least one State.

EP indicates a start point of the scenario. When an instruction of anexecution start of the scenario is issued from the user, a process forexecuting the scenario starts from EP.

Behavior and Event are described in each State.

Actions executed by respective apparatuses connected to the controller13 (center control unit 31) and corresponding to action control targetsare described in Behavior.

Action results of the control targets as conditions for shifting todifferent states as triggers for shifting to the different states, andstate shift destinations associated with the shift conditions aredescribed in Event.

Accordingly, in a case where an action executed by any one of therespective control targets meets a shift condition described in Event ofcertain State during execution of actions described in Behavior of thisState by the respective control targets, this State shifts to differentState described in association with the shift condition. The scenario istherefore considered to cover expressions of all shifts from States toStates based on Event.

Note that each of the shift conditions is assumed to be set on the basisof a change of a situation of the object in the lecturer camera image(lecturer L1) or the object in the auditor camera image (auditor A1).While details will be described below, examples of the shift conditionsinclude a condition based on movement of the lecturer L1, and acondition based on actions of the auditors A1.

An example in FIG. 5 depicts one EP and six States A to F. RespectiveStates are connected by unidirectional or bidirectional arrows. Thestate of the imaging system shifts between EP and respective States inaccordance with these arrows.

FIGS. 6 to 8 depict details of States A to F.

Behavior of each of States describes actions executed by respectiveresources (respective apparatuses provided as the lecturer cameracontroller 12-1, the auditor camera controller 12-2, and the switcher33) connected to the center control unit 31.

Concerning the lecturer camera controller 12-1 and the auditor cameracontroller 12-2, an action (upper row) associated with imaging controlby the corresponding camera 11, and calculation (lower row) performed bythe corresponding camera controller 12 are described. Concerning theswitcher 33, an action associated with image output is described.

The contents and the number of the actions of the respective apparatusesin Behavior vary in accordance with the contents and the number of thefunctions included in the respective apparatuses. Accordingly, thecenter control unit 31 is configured to inquire of the respectiveconnected apparatuses to recognize the contents and the number of thefunctions of the respective apparatuses.

Action results of the resources (apparatuses) corresponding to acondition of shifting to a subsequent state, and a state shiftdestination are described in Event in each of States. Only one shiftdestination and one shift condition, or a plurality of destinations anda plurality of conditions are described in each Event. In addition,Event may be described in common to respective States, rather than foreach State. For example, as Event in common to respective States, one ora plurality of conditions for forcibly shifting to a different state insuch cases as a case where an error is caused, and a case where aninterruption is made by the user in any one of States, and one or aplurality of shift destinations of the corresponding condition orconditions can be described.

As actions of the lecturer camera controller 12-1, fixed imaging on theWide side, and execution of moving body detection at the preset position(platform) (by the lecturer camera 11-1) are described in Behavior ofState A (FIG. 6). Moreover, as an action of the auditor cameracontroller 12-2, fixed imaging on the Wide side (by the auditor camera11-2) is described. Furthermore, as an action of the switcher 33, outputof the lecturer camera image is described.

A shift of the state to State B in a case where the lecturer cameracontroller 12-1 detects a moving body at the preset position (platform)is described in Event of State A.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State B (FIG. 6). Moreover, as actions ofthe auditor camera controller 12-2, fixed imaging on the Wide side, andexecution of gesture (standing up) detection (by the auditor camera11-2) are described. Furthermore, as an action of the switcher 33,output of the lecturer camera image is described.

A shift of the state to State C in a case where the auditor cameracontroller 12-2 detects a gesture (standing up) is described in Event ofState B.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State C (FIG. 7). Moreover, as actions ofthe auditor camera controller 12-2, pan-tilt imaging on the Tele side,and tracking and execution of gesture (sitting down) detection (by theauditor camera 11-2) are described. Furthermore, as an action of theswitcher 33, output of the auditor camera image is described.

A shift of the state to State D in a case where the auditor cameracontroller 12-2 determines that a tracking target shifts only by adistance shorter than a fixed distance for a fixed period or longer isdescribed in Event of State C. Moreover, a shift of the state to State Ein a case where the auditor camera controller 12-2 determines that atracking target does not shift by a fixed distance or longer in a periodshorter than a fixed period is described. Furthermore, a shift of thestate to State B in a case where the auditor camera controller 12-2detects a gesture (sitting down) in a period shorter than a fixed periodis described.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State D (FIG. 7). Moreover, as actions ofthe auditor camera controller 12-2, pan-tilt imaging on the Tele side,and execution of gesture (sitting down) detection (by the auditor camera11-2) are described. Furthermore, as an action of the switcher 33, it isdescribed that the lecturer camera image and the auditor camera imageare merged side by side to be outputted.

A shift of the state to State B in a case where the auditor cameracontroller 12-2 detects a gesture (sitting down) is described in Eventof State D.

As actions of the lecturer camera controller 12-1, fixed imaging on theTele side, and execution of moving body detection (by the lecturercamera 11-1) are described in Behavior of State E (FIG. 8). Moreover, asactions of the auditor camera controller 12-2, pan-tilt imaging on theTele side, and execution of tracking (by the auditor camera 11-2) aredescribed. Furthermore, as an action of the switcher 33, output of theauditor camera image is described.

A shift of the state to State F in a case where the lecturer cameracontroller 12-1 detects a plurality of moving bodies (i.e., two or morepersons as a lecturer and an auditor) at the preset position (platform)is described in Event of State E.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State F (FIG. 8). The pan-tilt imaging andtracking herein are performed in correspondence with a plurality ofpersons. Moreover, as an action of the auditor camera controller 12-2,fixed imaging on the Wide side (by the auditor camera 11-2) isdescribed. Furthermore, as an action of the switcher 33, output of thelecturer camera image is described.

A shift of the state to State B in a case where the lecturer cameracontroller 12-1 determines that the auditor has moved to the outside ofthe preset position (platform) is described in Event of State F.

Moreover, as Event common to respective States (FIG. 8), an end of theaction of the entire imaging system in a case where an end instructionis given to the center control unit 31 by the user is described.Furthermore, a shift of the state to State B in a case where an errorhas been caused in any one of all the resources (apparatuses)constituting the imaging system is described.

(Action Example of Imaging System Based on Scenario)

An action example of the imaging system based on the scenario describedabove will be hereinafter described.

Note that the action of the imaging system based on the scenarioaccording to the present embodiment may be executed only in a case wherea specific imaging mode is set. For example, the specific imaging modemay be set in response to a trigger of detection of the objects (each ofthe lecturer L1 and the auditor A1) in the camera images, or may be setin accordance with an operation by the user. Alternatively, the specificimaging mode may be set in response to a trigger of a lecture start timeset beforehand.

FIG. 9 depicts a scene from a time when the lecturer L1 stands on theplatform to a time when one of the auditors A1 stands up to ask aquestion in the imaging system in FIG. 1, and corresponds to a shiftfrom State A to State C of the scenario described above. As depicted inFIG. 9, note that the lecturer camera 11-1 in this example is disposedin a rear central part of the classroom where a lecture is given toimage the lecturer L1, while the auditor camera 11-2 is disposed in afront right part of the classroom to image the entire auditors A1.

In the figures each depicting an action example of the imaging systembased on the scenario, it is hereinafter assumed that the upper rowdepicts a top view of the classroom where a lecture is given for eachState, and that the middle and lower rows depict a lecturer camera imageand an auditor camera image, respectively, for each State. The top viewof the classroom depicts angles of view (imaging ranges) of the lecturercamera 11-1 and the auditor camera 11-2 each indicated by a broken line,and positions of the lecturer L1 and the auditor A1. In addition, animage surrounded by a thick frame and connected by a thick broken arrowin each of the lecturer camera image and the auditor camera imageindicates an image outputted from the switcher 33.

Initially, in response to an instruction of execution of the scenario isissued from the user to the controller 13 (center control unit 31) viathe input/output apparatus 14, the state shifts from EP (Entry Point) toState A. At this time, the center control unit 31 instructs the lecturercamera controller 12-1, the auditor camera controller 12-2, and theswitcher 33 to perform actions described in Behavior of State A (FIG.6).

In response to this instruction, the lecturer camera controller 12-1issues a camera control signal to instruct the lecturer camera 11-1 toperform fixed imaging on the Wide side, and also executes moving bodydetection in a state that the preset position is set to the platform inthe lecturer camera image as depicted in State A in FIG. 9.

In addition, the auditor camera controller 12-2 issues a camera controlsignal to instruct the auditor camera 11-2 to perform fixed imaging onthe Wide side.

Thereafter, the switcher 33 outputs the lecturer camera image surroundedby the thick frame.

When the lecturer camera controller 12-1 detects the moving body(lecturer L1) at the preset position (platform) in the lecturer cameraimage in this state, the state shifts to State B as described in Eventof State A (FIG. 6). At this time, the center control unit 31 instructsthe lecturer camera controller 12-1, the auditor camera controller 12-2,and the switcher 33 to perform actions described in Behavior of State B(FIG. 6).

In response to this instruction, the lecturer camera controller 12-1executes tracking of the lecturer L1 in the lecturer camera image asindicated in State B in FIG. 9. The lecturer camera controller 12-1 alsoissues a camera control signal to instruct the lecturer camera 11-1 toperform pan-tilt imaging on the Tele side to capture the lecturer L1 atan appropriate angle of view on the basis of a tracking result. Theappropriate angle of view herein is an angle of view for capturing abust-up shot of the lecturer L1.

In addition, the auditor camera controller 12-2 issues a camera controlsignal to instruct the auditor camera 11-2 to perform fixed imaging onthe Wide side continuously from State A, and executes gesture (standingup) detection in the auditor camera image.

Thereafter, the switcher 33 outputs the lecturer camera image surroundedby the thick frame continuously from State A.

When the auditor camera controller 12-2 detects a gesture (standing up)in the auditor camera image in this state, the state shifts to State Cas described in Event of State B (FIG. 6). At this time, the centercontrol unit 31 instructs the lecturer camera controller 12-1, theauditor camera controller 12-2, and the switcher 33 to perform actionsdescribed in Behavior of State C (FIG. 7) .

In response to this instruction, the lecturer camera controller 12-1continues an action similar to the action in State B as depicted inState C in FIG. 9.

In addition, the auditor camera controller 12-2 executes tracking of theauditor A1 in the auditor camera image. The auditor camera controller12-2 also issues a camera control signal to instruct the auditor camera11-2 to perform pan-tilt imaging on the Tele side to capture the auditorA1 at an appropriate angle of view on the basis of a tracking result.The appropriate angle of view herein is an angle of view for capturing abust-up shot of the auditor A1. In addition, the auditor cameracontroller 12-2 executes gesture (sitting down) detection of thestanding auditor A1 in the auditor camera image.

Thereafter, the switcher 33 switches the lecturer camera image to theauditor camera image surrounded by the thick frame, and outputs thisauditor camera image.

This state shifts to any one of State D, State E, or State B inaccordance with an action result of the auditor camera controller 12-2as described in Event of State C (FIG. 7).

FIG. 10 depicts a scene from a time when the auditor A1 stands up to atime when the auditor A1 answers a question on the spot and sits down inthe imaging system in FIG. 1, and corresponds to a shift in the order ofState C, State D, and State B of the scenario described above.

More specifically, in State C in FIG. 10, the state shifts to State D ina case where the auditor camera controller 12-2 determines that atracking target moves only by a distance shorter than a fixed distancefor a fixed period or longer (the auditor A1 answers a question whilestanding up at the seat) in the auditor camera image as described inEvent of State C (FIG. 7). At this time, the center control unit 31instructs the lecturer camera controller 12-1, the auditor cameracontroller 12-2, and the switcher 33 to perform actions described inBehavior of State D (FIG. 7).

In response to this instruction, the lecturer camera controller 12-1continues an action similar to the action in State C as depicted inState D in FIG. 10.

In addition, the auditor camera controller 12-2 executes gesture(sitting down) detection of the standing auditor A1 in the auditorcamera image.

Thereafter, the switcher 33 merges the lecturer camera image and theauditor camera image side-by-side and outputs the merged image. In thismanner, the audience can check an exchange between the lecturer L1 andthe auditor A1 in one screen even though the lecturer L1 and the auditorA1 are imaged by the different cameras 11.

When the auditor camera controller 12-2 detects a gesture (sitting down)in the auditor camera image in this state, the state again shifts toState B as described in Event of State D (FIG. 7). In response to thisshift, the switcher 33 switches the lecturer camera image and theauditor camera image from the image obtained by side-by-side merging tothe lecturer camera image surrounded by the thick frame, and outputsthis lecturer camera image as depicted in State B in FIG. 10.

FIG. 11 depicts a scene from a time when the auditor A1 stands up, movesto the platform, and answers a question to a time when the auditor A1returns to the seat and sits down in the imaging system in FIG. 1, andcorresponds to a shift in the order of State C, State E, State F, andState B of the scenario described above.

More specifically, in State C in FIG. 11, the state shifts to State E ina case where the auditor camera controller 12-2 determines that atracking target moves by a distance of a fixed distance or longer in aperiod shorter than a fixed period (the auditor moves to the platform toanswer a question) in the auditor camera image as described in Event ofState C (FIG. 7). At this time, the center control unit 31 instructs thelecturer camera controller 12-1, the auditor camera controller 12-2, andthe switcher 33 to perform actions described in Behavior of State E(FIG. 8).

In this manner, as depicted in State E in FIG. 11, the lecturer cameracontroller 12-1 issues a camera control signal to instruct the lecturercamera 11-1 to perform pan-tilt imaging on the Wide side. In addition,the lecturer camera controller 12-1 executes moving body detection inthe lecturer camera image. Even in a case where not only the lecturer L1but also the auditor A1 stands on the platform, both the auditor A1 andthe lecturer L1 can be captured by pan-tilt imaging on the Wide sideusing the lecturer camera 11-1.

In addition, the auditor camera controller 12-2 issues a camera controlsignal to instruct the auditor camera 11-2 to perform pan-tilt imagingon the Tele side continuously from State C, and executes tracking of theauditor A1 in the auditor camera image.

Thereafter, the switcher 33 outputs the auditor camera image surroundedby the thick frame continuously from State C.

When the lecturer camera controller 12-1 detects a plurality of themoving bodies (i.e., two persons as the lecturer L1 and the auditor A1)at the preset position (platform) in the lecturer camera image in thisstate, the state shifts to State F as described in Event of State E(FIG. 8). At this time, the center control unit 31 instructs thelecturer camera controller 12-1, the auditor camera controller 12-2, andthe switcher 33 to perform actions described in Behavior of State F(FIG. 8).

In response to this instruction, the lecturer camera controller 12-1executes tracking of each of the lecturer L1 and the auditor A1 in thelecturer camera image as indicated in State F in FIG. 11. The lecturercamera controller 12-1 also issues a camera control signal to instructthe lecturer camera 11-1 to perform pan-tilt imaging on the Tele side tocapture the objects at an appropriate angle of view on the basis of atracking result. The appropriate angle of view herein is an angle ofview for capturing both the lecturer L1 the auditor A1.

In addition, the auditor camera controller 12-2 issues a camera controlsignal to instruct the auditor camera 11-2 to perform fixed imaging onthe Wide side to have the same angle of view as that angle of State B asa subsequent shift destination. Moving body detection and tracking arenot executed herein.

Thereafter, the switcher 33 switches the auditor camera image to thelecturer camera image surrounded by the thick frame, and outputs thislecturer camera image.

When the lecturer camera controller 12-1 determines that the auditor hasmoved to the outside of the preset position (platform) in the lecturercamera image in this state, the state again shifts to State B asdescribed in Event of State F (FIG. 8). In response to this shift, theswitcher 33 outputs the lecturer camera image surrounded by the thickframe continuously from State F as depicted in State B in FIG. 11.

According to the configuration described above, tracking of a lecturerand detection of an action of an auditor such as standing up and sittingdown, and also capturing the lecturer and the auditor at an appropriateangle of view are achievable using only two cameras. More specifically,one of the lecturer camera and the auditor camera zooms in to capture animage an object, while the other camera zooms out to detect the objectwith control linkage between the two cameras. The respective images areswitchable in appropriate timing and output.

Accordingly, providable is such a lecture capture system which achievescost reduction by eliminating the necessity of providing an overheadcamera for detecting an object in addition to a tracking camera fortracking and imaging an object.

(Scenario Editing Example)

The scenario described above is editable by the user.

According to the scenario described above, the image to be outputted isswitched from the lecturer camera image to the auditor camera image(State B→State C) in a case where the auditor A1 stands up to ask aquestion as described with reference to FIG. 9.

By allowing editing of the scenario for the foregoing action of theimaging system, the image to be outputted when the auditor A1 raises hisor her hand to ask a question is allowed to switch from the lecturercamera image to the auditor camera image as depicted in FIG. 12 (StateB′→State G).

FIG. 13 is a diagram depicting a description example of the scenarioafter editing.

According to the scenario in FIG. 13, State B in the scenario in FIG. 5is switched to State B′, and State G is newly added.

FIG. 14 depicts details of State B′ and State G.

As an action of the auditor camera controller 12-2 to be described inBehavior of State B′, “execute gesture (standing up) detection”described in Behavior of State B (FIG. 6) is switched to “executegesture (raising hand) detection.”

In addition, as a shift to be described in Event of State B′ in a casewhere the auditor camera controller 12-2 detects a gesture (standingup), a shift to “State C” is switched to a shift to “state G.”

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State G newly added. Moreover, as actionsof the auditor camera controller 12-2, fixed imaging on the Wide side,and execution of gesture (standing up) detection (by the auditor camera11-2) are described. Furthermore, as an action of the switcher 33,output of the auditor camera image is described.

A shift of the state to State C in a case where the auditor cameracontroller 12-2 detects a gesture (standing up) is described in Event ofState G.

More specifically, when the auditor camera controller 12-2 detects agesture (raising hand) in the auditor camera image in State B′ in FIG.12, the state shifts to State G as described in Event of State B′ (FIG.14). At this time, the center control unit 31 instructs the lecturercamera controller 12-1, the auditor camera controller 12-2, and theswitcher 33 to perform actions described in Behavior of State G (FIG.14).

In response to this instruction, the lecturer camera controller 12-1continues an action similar to the action in State B′ as depicted inState G in FIG. 12.

In addition, the auditor camera controller 12-2 issues a camera controlsignal to instruct the auditor camera 11-2 to perform fixed imaging onthe Wide side continuously from State B′, and executes gesture (standingup) detection in the auditor camera image.

Thereafter, the switcher 33 switches the lecturer camera image to thelecturer camera image surrounded by the thick frame, and outputs thislecturer camera image.

When the auditor camera controller 12-2 detects a gesture (standing up)in the auditor camera image in this state as described in Event of StateG (FIG. 14), the state shifts to State C. In response to this shift, theswitcher 33 outputs the lecturer camera image surrounded by the thickframe continuously from State G as indicated in State C in FIG. 12.

The action of the entire imaging system can be changed by editing thescenario in the manner described above.

(Example of GUI used for Editing of Scenario)

Editing of the scenario in the manner described above is achieved bypresenting GUI (Graphical User Interface) to the user using thecontroller 13 (center control unit 31). The controller 13 (centercontrol unit 31) can add or delete States, and edit (change) States(description contents of Behaviors and Events) included in the existingscenario in accordance with an operation performed by the user through akeyboard or a mouse provided as the input/output apparatus 14.

FIG. 15 is a diagram depicting an example of an editing screen of thescenario displayed as GUI used for editing of the scenario in a displayprovided as the input/output apparatus 14.

An editing screen of the scenario described with reference to FIG. 5 isdisplayed in a display 100 in FIG. 15.

In FIG. 15, a context menu 111 is displayed in response to a right clickoperation or the like in a state where State B constituting the scenariois selected by the mouse or the like. The context menu 111 displaysthree items, i.e., “add State,” “delete State,” and “edit State.” Atriangular arrow is displayed at the right end of the item “edit State,”and a sub menu is displayed on the right side of the arrow. Two items,i.e., “edit Behavior” and “edit Event” are displayed in the sub menu.

If “edit Behavior” of the sub menu is selected in FIG. 15, a pop-upwindow 112 for editing Behavior of State B is displayed in the editingscreen as depicted in FIG. 16. As Behavior of State B, a list describingcontents of actions of the respective apparatuses is displayed in thepop-up window 112 in FIG. 16.

A pull-down menu in FIG. 16 is displayed by selecting an item portion ofan action performed by the auditor camera controller 12-2 (executegesture (standing up) detection) using the mouse or the like. Actionsexecutable by the auditor camera controller 12-2, such as “executemoving body detection,” “execute tracking,” “execute gesture (raisinghand) detection,” and “execute gesture (sitting down) detection” aredisplayed in the pull-down menu as choices.

The choices displayed in the pull-down menu are actions executable inaccordance with the contents and the number of the functions of therespective apparatuses of which the controller 13 (center control unit31) inquires regarding these contents and number. Moreover, the choicesto be displayed in the pull-down menu may be manually added by the useron the basis of specifications of the respective apparatuses.

Note that the editing screen of the scenario may be such a screen whichincludes a menu button group 121 displayed in a lower right region ofthe screen in response to a right click operation or the like in theediting screen using the mouse as depicted in FIG. 17, for example, aswell as the example depicted in FIG. 15. Alternatively, the menu buttongroup 121 may be displayed as default at a fixed position in the editingscreen of the scenario.

Furthermore, the user can expand the functions of the respectiveapparatuses. For example, the user can newly define and register typesof gesture detected by the gesture detection unit 22 of the cameracontroller 12.

FIG. 18 is a diagram depicting an example of a gesture registrationscreen as GUI used for defining and registering gestures, and displayedin the display provided as the input/output apparatus 14.

FIG. 18 depicts a state of registration of a gesture corresponding to astanding-up action in a gesture registration screen 150. The gestureregistration screen 150 is displayed in response to selection of agesture definition and registration mode by the user using software, forexample.

An image display section 151 is included in the gesture registrationscreen 150. An image of any object (person) captured by the cameras 11is displayed in the image display section 151. A person in a sittingstate appears in the image display section 151 of the gestureregistration screen 150 on the left side in FIG. 18, while the person ina standing state shifted from the sitting state appears in the imagedisplay section 151 of the gesture registration screen 150 on the rightside in FIG. 18.

The gesture detection unit 22 of the camera controller 12 recognizes aposture of the person in the image, generates skeletal structureinformation indicated by thick lines in the image display section 151,and supplies the generated skeletal structure information to the centercontrol unit 31 via the camera control unit 24. The switcher 33superimposes the skeletal structure information acquired by the centercontrol unit 31 on an image received from the camera controller 12, anddisplays the superimposed information in the image display section 151of the gesture registration screen 150. The skeletal structureinformation may be superimposed on the image at the camera controller12.

Buttons b1, b2, b3, and b4, and a text box tb are provided above theimage display section 151 in the gesture registration screen 150.

The button b1 is a button for starting recording of the image displayedin the image display section 151, while the button b2 is a button forstopping recording of the image. The button b3 is a button forreproducing the recorded image, while the button b4 is a button forediting skeletal structure information in the reproduced image. The usercan correct the skeletal structure information in the image reproducedand displayed in the image display section 151 by operating the buttonb4 using the mouse or the like.

The text box tb is a region to which a name of a gesture to beregistered is inputted. The user can determine a name of a gesture to beregistered by operating the keyboard or the like.

A timeline 152 and a registration button 153 are provided below theimage display section 151 in the gesture registration screen 150.

The timeline 152 is an object which indicates a reproduction position ofthe image displayed in the image display section 151. A cursor Rc on abar indicates a current reproduction position in the entire image interms of time.

In the recorded image, an icon Rs on the bar of the timeline 152represents a start time of an image actually used for gesture detectionin terms of time, and an icon Re represents an end time of the imageactually used for gesture detection in terms of time. The positions ofthe icons Rs and Re on the bar of the timeline 152 can be set by theuser.

The registration button 153 is a button for registering the skeletalstructure information included in the recorded image in association witha gesture name. A series of changes of the skeletal structureinformation included in the image from the start position indicated bythe icon Rs to the end position indicated by the icon Re in the recordedimage are registered by operating the registration button 153 togetherwith the gesture name inputted to the text box tb.

The gesture name registered in the above manner is added to anddisplayed in the pull-down menu in FIG. 16. The gesture detection unit22 can detect a gesture identical or similar to a newly registeredgesture by changing the action of the camera controller 12 to executionof detection of the added gesture during editing of the scenario(Behavior).

Note that movement of the object in the camera image and the skeletalstructure information included in the recorded image may be comparedwith each other in time series, or in units of predetermined frame inthe gesture detection performed by the gesture detection unit 22.

According to the example described in FIG. 18, the standing-up action isregistered as the gesture. However, in view of the fact that a series ofposture changes are recordable, any gesture is allowed to be registered.

Moreover, while the example of gesture registration is described herein,board extraction patterns employed by a board extraction unit 211 (FIG.20) described below, and faces used for identification by a facedetection/identification unit 351 described below may be similarlyregistered.

According to the configuration described above, the user can implement asoftware structure and GUI capable of defining and editing the actionsof the entire system. More specifically, the user can add and deleteaction procedures of the system by combining States, or change shiftsbetween States in the scenario which describes the actions (Behaviors)of respective resources defining States, and Events for shiftingsubsequent States.

Accordingly, the user (lecturer) is allowed to give lectures inaccordance with a style of each lecture, and flexibly cope with changesof the system configuration.

2. SECOND EMBODIMENT (Configuration Example of Imaging System)

FIG. 19 is a diagram depicting another configuration example of theimaging system to which the present technology is applied.

An imaging system in FIG. 19 includes a board camera 11-3 and a boardcamera controller 12-3 in addition to the configuration of the imagingsystem in FIG. 1.

The board camera 11-3 captures an image of a blackboard (or whiteboard)B1 used by the lecturer L1 during a lecture under control by the boardcamera controller 12-3, and outputs the obtained image (board cameraimage) to the board camera controller 12-3.

The board camera 11-3 is provided in such a position as to imagesubstantially in the same direction as the direction of the lecturercamera 11-1.

The board camera 11-3 may have a configuration similar to theconfigurations of the lecturer camera 11-1 and the auditor camera 11-2except that an object corresponding to an imaging target is theblackboard B1. Accordingly, similarly to the lecturer camera 11-1 andthe auditor camera 11-2, the board camera 11-3 is also configured as aPTZ camera which has a function of mechanical panning and tilting, andalso is capable of performing optical and electronical zooming.

The board camera controller 12-3 controls the board camera 11-3 undercontrol by the controller 13 to acquire a board camera image outputtedfrom the board camera 11-3, and outputs the acquired image to thecontroller 13. In addition, the board camera controller 12-3 receives aninstruction of calculation and control from the controller 13, andoutputs a result of calculation and control to the controller 13.

The board camera controller 12-3 basically has a configuration similarto the configurations of the lecturer camera controller 12-1 and theauditor camera controller 12-2 except that the control target is theboard camera 11-3, but is different in that a board extraction functiondescribed below is provided.

(Functional Configuration Example of Board Camera Controller)

FIG. 20 is a block diagram depicting a functional configuration exampleof the board camera controller 12-3 described above.

The board camera controller 12-3 in FIG. 20 includes the boardextraction unit 211 in addition to the configuration of the cameracontroller 12 in FIG. 2.

The board extraction unit 211 acquires a board camera image from theboard camera 11-3, and executes a board extraction process forextracting contents written on the blackboard B1 from the board cameraimage under control by the camera control unit 24. While details of theboard extraction process are not described herein, a board extractionresult image resulting from the board extraction process is outputted tothe controller 13 (switcher 33) as an image signal. In this case, theswitcher 33 controls output of at least either respective camera imagesor the board extraction result image in accordance with the image of theobject in each of the camera images, or the result of the boardextraction process under control by the center control unit 31.

The board camera image includes not only the blackboard B1, but also thelecturer L1 who writes characters, graphs or the like on the blackboardB1. The board extraction unit 211 outputs the board extraction imagewhile deleting the lecturer L1 from the board camera image, andincreasing visibility of the characters, graphs or the like written onthe board by reproducing and emphasizing these characters or the like.

Moreover, the board extraction unit 211 is capable of not onlyextracting the contents written on the board, but also storing a processwhere the lecturer L1 writes or deletes characters on the blackboard B1in time series for recognition of timing of a writing start or adeleting start, and the written contents (characters, figures or thelike).

(Description Example of Scenario)

FIG. 21 is a diagram depicting a description example of a scenario whichdefines actions of the imaging system in FIG. 19.

According to the scenario in FIG. 21, State B′ in the scenario in FIG.13 is switched to State B″, and States H and I are newly added.

FIG. 22 is a diagram depicting details of State B″, while FIG. 23 is adiagram depicting details of States H and I.

Actions of the board camera controller 12-3 are described in Behavior ofState B″ (FIG. 22) in addition to description similar to Behavior ofState B′ (FIG. 13). As the actions of the board camera controller 12-3,fixed imaging on the Wide side, and execution of board extraction(process) (by the board camera 11-3) are described in Behavior of StateB″.

A shift of the state to State H in a case where the board cameracontroller 12-3 detects writing to or deleting from the board isdescribed in Event of State B″ in addition to description similar toBehavior of State B′ (FIG. 13).

As actions of the lecturer camera controller 12-1, fixed imaging on theWide side, and execution of tracking (by the lecturer camera 11-1) aredescribed in Behavior of State H (FIG. 23). Moreover, as actions of theauditor camera controller 12-2, fixed imaging on the Wide side, andexecution of gesture (raising hand) detection (by the auditor camera11-2) are described. Furthermore, as the actions of the board cameracontroller 12-3, fixed imaging on the Wide side, and execution of boardextraction (by the board camera 11-3) are described. As an action of theswitcher 33, output of a board extraction result image is described.

A shift of the state to State G in a case where the auditor cameracontroller 12-2 detects a gesture (raising hand) is described in Eventof State H. Moreover, a shift of the state to State I in a case wherethe board camera controller 12-3 determines that writing to or deletingfrom the board is not detected for a fixed period or longer isdescribed.

As actions of the lecturer camera controller 12-1, fixed imaging on theTele side, and execution of tracking (by the lecturer camera 11-1) aredescribed in Behavior of State I (FIG. 23). Moreover, as actions of theauditor camera controller 12-2, fixed imaging on the Wide side, andexecution of gesture (raising hand) detection (by the auditor camera11-2) are described. Furthermore, as the actions of the board cameracontroller 12-3, fixed imaging on the Wide side, and execution of boardextraction (by the board camera 11-3) are described. As an action of theswitcher 33, it is described that the lecturer camera image and theboard extraction result image are merged side-by-side to be outputted.

A shift of the state to State G in a case where the auditor cameracontroller 12-2 detects a gesture (raising hand) is described in Eventof State I. Moreover, a shift of the state to State H in a case wherethe board camera controller 12-3 detects writing to or deleting from theboard is described.

(Action Example of Imaging System Based on Scenario)

Subsequently, an action example of the imaging system based on thescenario in FIG. 21 will be described.

The scenario in FIG. 21 is different from the scenario in FIG. 13 inthat State B″, State H, and State I are provided. Accordingly, stateshifts between these States will be touched upon with reference to FIG.24.

When the auditor camera controller 12-2 detects a gesture (raising hand)in the auditor camera image in State B″ in FIG. 24, the state shifts toState G as described in Event of State B″ (FIG. 22). In response to thisshift, the switcher 33 switches the lecturer camera image to the auditorcamera image, and outputs the auditor camera image as indicated in StateG in FIG. 24.

On the other hand, when the board camera controller 12-3 detects writingto or deleting from the board in the board camera image, the stateshifts to State H as described in Event of State B″ (FIG. 22). At thistime, the center control unit 31 instructs the lecturer cameracontroller 12-1, the auditor camera controller 12-2, the board cameracontroller 12-3, and the switcher 33 to perform actions described inBehavior of State H (FIG. 23).

In response to this instruction, the switcher 33 switches the lecturercamera image to the board extraction result image, and outputs the boardextraction result image as indicated in State H in FIG. 24.

When the auditor camera controller 12-2 detects a gesture (raising hand)in the auditor camera image in this state, the state shifts to State Gas described in Event of State H (FIG. 23). In response to this shift,the switcher 33 switches the board extraction result image to theauditor camera image, and outputs the auditor camera image as indicatedin State G in FIG. 24.

On the other hand, when the board camera controller 12-3 determines thatwriting to or deleting from the board is not detected in the boardcamera image for a fixed period or longer, the state shifts to State Ias described in Event of State H (FIG. 23). At this time, the centercontrol unit 31 instructs the lecturer camera controller 12-1, theauditor camera controller 12-2, the board camera controller 12-3, andthe switcher 33 to perform actions described in Behavior of State I(FIG. 23).

In response to this instruction, the switcher 33 switches from the boardextraction result image and output the image obtained by merging thelecturer camera image and the board extraction result image side-by-sideas indicated in State I in FIG. 24.

When the auditor camera controller 12-2 detects a gesture (raising hand)in the auditor camera image in this state, the state shifts to State Gas described in Event of State I (FIG. 23). In response to this shift,the switcher 33 switches the lecturer camera image and the boardextraction result image obtained by side-by-side merging to the auditorcamera image, and outputs the auditor camera image as depicted in StateG in FIG. 24.

On the other hand, when the board camera controller 12-3 detects writingto or deleting from the board, the state again shifts to State H asdescribed in Event of State H (FIG. 23). In response to this shift, theswitcher 33 switches the lecturer camera image and the board extractionresult image obtained by side-by-side merging to the board extractionresult image, and outputs the board extraction result image as depictedin State H in FIG. 24.

According to the configuration described above, the board extractionresult image is outputted during writing on the blackboard B1 by thelecturer L1 (State H). When the lecturer L1 stops board writing for awhile, an image in which a state of the lecturer L1 appears is outputtedin addition to the board extraction result image (State I). Basically,the shift between State H and State I is repeated as the state of theimaging system to output an image in which writing on the board chieflyappears. Accordingly, the audience can recognize detailed contents ofthe lecture.

3. THIRD EMBODIMENT (Configuration Example of Imaging System)

FIG. 25 is a diagram depicting a further configuration example of theimaging system to which the present technology is applied.

The imaging system in FIG. 25 includes a PC for slide 301 in addition tothe configuration of the imaging system in FIG. 1.

The PC for slide 301 as an information processing apparatus is connectedto the controller 13, and outputs slides for presentation and dataassociated with materials to the controller 13 under control by thecontroller 13. In this case, the switcher 33, in this case, the switcher33 controls output of at least either the respective camera images orthe slides in accordance with a situation of the object of each of thecamera images (including output change of the slides (materials)). Notethat the slides and the materials outputted from the PC for slide 301are also output to (displayed in) a not-depicted screen provided behindthe lecturer L1 (on the side opposite to the auditors A1).

The PC for slide 301 is configured as a personal computer, but may beconfigured as a portable terminal apparatus such as a tablet terminaland a smartphone.

(Description Example of Scenario)

FIG. 26 is a diagram depicting a description example of a scenario whichdefines actions of the imaging system in FIG. 25.

According to the scenario in FIG. 26, State B in the scenario in FIG. 5is switched to State B′″, and States J, K, L, and M are provided insteadof States C, D, E, and F.

FIG. 27 is a diagram depicting details of State B′″, while FIGS. 28 and29 are diagrams depicting details of States J, K, L, and M.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State B′″ (FIG. 27). Moreover, as actionsof the auditor camera controller 12-2, fixed imaging on the Wide side,and execution of gesture (applause) detection (by the auditor camera11-2) are described. Further described are output of the slides as anaction of the PC for slide 301, and output of the lecturer camera imageas an action of the switcher 33.

A shift of the state to State M in a case where the auditor cameracontroller 12-2 detects a gesture (applause) is described in Event ofState B′″. Moreover, a shift of the state to State J in a case where thePC for slide 301 executes switching of slides (slide feeding) isdescribed.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described as Behavior of State J (FIG. 28). Moreover, as actions ofthe auditor camera controller 12-2, fixed imaging on the Wide side, andexecution of gesture (applause) detection (by the auditor camera 11-2)are described. Further described are output of slides as an action ofthe PC for slide 301, and output of these slides as an action of theswitcher 33.

A shift of the state to State K in a case where the lecturer cameracontroller 12-1 detects a gesture (body language), and a shift of thestate to State L in a case where the lecturer camera controller 12-1determines that the tracking target has moved by a fixed distance orlonger are described in Event of State J. Moreover, a shift of the stateto State M in a case where the auditor camera controller 12-2 detects agesture (applause) is described.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State K (FIG. 28). Moreover, as actions ofthe auditor camera controller 12-2, fixed imaging on the Wide side, andexecution of gesture (applause) detection (by the auditor camera 11-2)are described. Further described are output of the slides as an actionof the PC for slide 301, and side-by-side merging of the lecturer cameraimage and the slides and outputting the merged image as an action of theswitcher 33.

A shift of the state to State L in a case where the lecturer cameracontroller 12-1 determines that the tracking target has moved by a fixeddistance or longer is described in Event of State K. Moreover, a shiftof the state to State M in a case where the auditor camera controller12-2 detects a gesture (applause) is described. Furthermore, a shift ofthe state to State J in a case where the PC for slide 301 executesswitching of the slides is described.

As actions of the lecturer camera controller 12-1, pan-tilt imaging atan angle of view sufficient for including both the lecturer L1 and theslides output to the screen behind the lecturer L1, and execution oftracking (by the lecturer camera 11-1) are described in Behavior ofState L (FIG. 29). Moreover, as actions of the auditor camera controller12-2, fixed imaging on the Wide side, and execution of gesture(applause) detection (by the auditor camera 11-2) are described. Furtherdescribed are output of the slides as an action of the PC for slide 301,and output of the lecturer camera image as an action of the switcher 33.

A shift of the state to State K in a case where the lecturer cameracontroller 12-1 detects a gesture (body language) is described in Eventof State L. Moreover, a shift of the state to State M in a case wherethe auditor camera controller 12-2 detects a gesture (applause) isdescribed. Furthermore, a shift of the state to State J in a case wherethe PC for slide 301 executes switching of the slides is described.

As actions of the lecturer camera controller 12-1, pan-tilt imaging onthe Tele side, and execution of tracking (by the lecturer camera 11-1)are described in Behavior of State M (FIG. 29). Moreover, as an actionof the auditor camera controller 12-2, pan-tilt imaging on the Tele side(by the auditor camera 11-2) is described. Further described are outputof the slides as an action of the PC for slide 301, and output of theauditor camera image as an action of the switcher 33.

A shift of the state to State B′″ in a case where the center controlunit 31 determines an elapse of a fixed period since the outputted imageis switched to the auditor camera image, for example, is described inEvent of State M.

(Action Example of Imaging System Based on Scenario)

Subsequently, an action example of the imaging system based on thescenario in FIG. 26 will be described.

The scenario in FIG. 26 is different from the scenario in FIG. 5 in thatState B′″, State J, State K, State L, and State M are provided.Accordingly, state shifts between these States will be described withreference to FIG. 30.

When the auditor camera controller 12-2 detects a gesture (applause) inthe auditor camera image in State B′″ in FIG. 30, the state shifts toState M as described in Event of State B′″ (FIG. 27).

On the other hand, the state shifts to State J when the PC for slide 301executes switching of the slides as described in Event of State B′″(FIG. 27). In response to this shift, the switcher 33 switches thelecturer camera image to the slides, and outputs the slides as indicatedin State J in FIG. 30.

When the lecturer camera controller 12-1 detects a gesture (bodylanguage) in the lecturer camera image in this state, the state shiftsto State K as described in Event of State J (FIG. 28). In response tothis shift, the switcher 33 switches from the slides and outputs animage obtained by merging the lecturer camera image and the slides sideby side as indicated in State K in FIG. 30.

When the lecturer camera controller 12-1 determines that the trackingtarget has moved by a fixed distance or longer in the lecturer cameraimage in this state, the state shifts to State L as described in Eventof State K (FIG. 28). In response to this shift, the switcher 33switches from the image obtained by merging the lecturer camera imageand the slides side by side and outputs the lecturer camera image at anangle of view sufficient for including the lecturer L1 and the slidesbehind the lecturer L1 as depicted in State L in FIG. 30.

When the lecturer camera controller 12-1 detects a gesture (bodylanguage) in the lecturer camera image in this state, the state shiftsto State K as described in Event of State L (FIG. 29). In addition, thestate again shifts to State J when the PC for slide 301 executesswitching of the slides.

On the other hand, when the auditor camera controller 12-2 detects agesture (applause) in the auditor camera image in State K, the stateshifts to State M as described in Event of State K (FIG. 28). Inaddition, the state again shifts to State J when the PC for slide 301executes switching of the slides.

Furthermore, the state again shifts to state L when the lecturer cameracontroller 12-1 determines that the tracking target has moved by a fixeddistance or longer in the lecturer camera image in State J as describedin Event of State J (FIG. 28). On the other hand, when the auditorcamera controller 12-2 detects a gesture (applause) in the auditorcamera image, the state shifts to State M.

In State M, the auditor camera image is outputted as indicated in StateM in FIG. 30. While execution of pan-tilt imaging on the Tele side isdescribed in Behavior of State M (FIG. 29), the entire auditors may beimaged by fixed imaging on the Wide side. Moreover, the imaging positionand camerawork of the auditor camera 11-2 in State M may be presetbeforehand.

Thereafter, when the center control unit 31 determines an elapse of afixed period such as completion of the preset camerawork, the stateagain shifts to State B′″ as described in Event of State M (FIG. 29).

In a case of slide switching, this slide is highly likely to be a slidethe lecturer L1 desires to show, and therefore is outputted according tothe configuration described above (State J). In a case where a bodylanguage performed by the lecturer L1 is exaggerated, an image in whicha state of the lecturer appears is outputted in addition to the slide(State K). In this manner, enthusiasm of the lecturer L1 for the lecturecan be effectively felt by the audience.

Moreover, in a case where the lecturer L1 starts walking, a dynamicimage can be provided by imaging with camerawork which tracks thelecturer L1 at a slightly zooming-out angle of view (State L).Furthermore, in a case where the auditors A1 applaud, an image with arealistic sensation can be provided by outputting an image in which theauditors appear (State M).

Basically, images in which the slides chiefly appear are outputted, andalso images of the lecturer L1 and the auditor A1 are effectivelyinserted. Accordingly, the audience can recognize detailed contents ofthe presentation while feeling a realistic sensation.

4. OTHER VARIATION EXAMPLES

Other variations of the embodiments described above will be hereinafterdescribed.

(Configuration Example of Imaging System)

FIG. 31 is a diagram depicting a still further configuration example ofthe imaging system to which the present technology is applied.

An imaging system in FIG. 31 includes the board camera 11-3, the boardcamera controller 12-3, and the PC for slide 301 in addition to theconfiguration of the imaging system in FIG. 1.

Accordingly, the imaging system in FIG. 31 has a configuration combiningthe imaging system in FIG. 19 and the imaging system in FIG. 25.

According to this configuration, the audience can recognize detailedcontents of presentation while recognizing detailed contents of alecture, or feeling a realistic sensation.

(Functional Configuration Example of Camera Controller)

FIG. 32 is a block diagram depicting another functional configurationexample of the camera controller 12 described above.

The board camera controller 12 in FIG. 32 includes the facedetection/identification unit 351 in addition to the configuration ofthe camera controller 12 in FIG. 2.

The face detection/identification unit 351 acquires a camera image fromeach of the cameras 11, and detects a face in the camera image oridentifies this face as a particular face under control by the cameracontrol unit 24. Information indicating that the face has been detected,or information indicating that the face has been identified is suppliedto the controller 13 (center control unit 31) via the camera controlunit 24.

Note that the face detection/identification unit 351 may detect a humanor identify this human as a particular person, rather than the face inthe camera image. Moreover, the camera controller 12 may additionallyhave a sophisticated function performing other types of recognitionprocessing, artificial intelligence processing or the like, rather thanthe face detection/identification unit 351.

In a case where the camera controller 12 includes the facedetection/identification unit 351, State A and State E constituting thescenario in FIG. 5 may be switched to State A′ and State E′,respectively, depicted in FIG. 33.

As the action of the lecturer camera controller 12-1 to be described inBehavior of State A′, “execute moving body detection at the presetposition (platform)” described in Behavior of State A (FIG. 6) isswitched to “execute face detection or face identification at the presetposition (platform).”

In addition, as the condition for shifting to State B to be described inEvent of State A′, “moving body is detected at the preset position(platform)” by the lecturer camera controller 12-1 is switched to “faceis detected or particular face is identified at the preset position(platform).”

On the other hand, as the action of the lecturer camera controller 12-1to be described in Behavior of State E′, “execute moving body detection”described in Behavior of State E (FIG. 8) is switched to “execute facedetection or face identification.”

In addition, as the condition for shifting to State F to be described inEvent of State E′, “a plurality of the moving bodies are detected at thepreset position (platform)” by the lecturer camera controller 12-1 isswitched to “a plurality of (i.e., two or more persons as lecturer andauditor) faces are detected or a plurality of particular faces areidentified at the preset position (platform).”

In this manner, the state of the imaging system can be shifted inresponse to a trigger given by the added function, such as facedetection and face identification. Moreover, the existing function andthe added function may be combined and executed to achieve anappropriate angle of view or camerawork, for example, by utilizing theadded function such as the position of the face detected by facedetection.

(Other Description Example of Scenario)

While the scenario is expressed by the state shift chart in the abovecase, the scenario may be expressed by a timeline as depicted in FIG.34.

The scenario in FIG. 34 is a scenario depicted in FIG. 5 and expressedby a timeline to represent a shift of the state in time series from theleft side to the right side in the figure. The scenario in FIG. 34indicates actions (Behaviors) executed by the lecturer camera controller12-1, the auditor camera controller 12-2, and the switcher 33 for eachState.

It is assumed in the scenario in FIG. 34 that the action of each of theapparatuses (resources) after State C is branched into three parts, andthat branching conditions in this case are similar to the correspondingconditions in Event of State C indicated in FIG. 7. Note that “Go to ★”refers to a return to a start point (★) of State B.

In this manner, the scenario may be expressed by a timeline as well.

(Configuration Example of Computer)

A series of processes described above may be executed by either hardwareor software. In a case where the series of processes are executed bysoftware, a program constituting the software is installed from aprogram recording medium into a computer incorporated in dedicatedhardware, a general-purpose personal computer, or the like.

FIG. 35 is a block diagram depicting a configuration example of hardwareof a computer which executes the series of processes described aboveunder the program.

The controller 13 described above is implemented by a computer having aconfiguration depicted in FIG. 35.

A CPU 1001, a ROM 1002, a RAM 1003 are connected to each other via a bus1004.

An input/output interface 1005 is further connected to the bus 1004. Aninput unit 1006 including a keyboard, a mouse and the like, and anoutput unit 1007 including a display, a speaker and the like areconnected to the input/output interface 1005. Moreover, a storage unit1008 including a hard disk, a non-volatile memory and the like, acommunication unit 1009 including a network interface and the like, anda drive 1010 driving a removable medium 1011 are connected to theinput/output interface 1005.

According to the computer configured as above, the CPU 1001 executes theprogram stored in the storage unit 1008 and loaded into the RAM 1003 viathe input/output interface 1005 and the bus 1004 to perform the seriesof processes described above, for example.

The program executed by the CPU 1001 is recorded in the removable medium1011, or provided via a wired or wireless transmission medium such as alocal area network, the Internet, and digital broadcasting, for example,and is installed in the storage unit 1008.

Note that the program executed by the computer may be a program whereprocesses are performed in time series in the order described in thepresent description, or may be a program where processes are performedin parallel or at necessary timing such as an occasion when a call ismade.

5. APPLICATION EXAMPLES

The technology according to the present disclosure is applicable tovarious systems. For example, the imaging system described above isapplicable to a conference system, or a surgery room system describedhereinafter.

FIG. 36 is a view schematically depicting a general configuration of asurgery room system 5100 to which the technology according to anembodiment of the present disclosure can be applied. Referring to FIG.36, the surgery room system 5100 is configured such that a group ofapparatus installed in a surgery room are connected for cooperation witheach other through an audiovisual (AV) controller 5107 and a surgeryroom controlling apparatus 5109.

In the surgery room, various apparatus may be installed. In FIG. 36, asan example, various apparatus group 5101 for endoscopic surgery, aceiling camera 5187, a surgery field camera 5189, a plurality of displayapparatus 5103A to 5103D, a recorder 5105, a patient bed 5183 and anillumination 5191 are depicted. The ceiling camera 5187 is provided onthe ceiling of a surgery room and images the hands of a surgeon. Thesurgery field camera 5189 is provided on the ceiling of the surgery roomand images a state of the entire surgery room.

Among the apparatus mentioned, the apparatus group 5101 belongs to anendoscopic surgery system 5113 hereinafter described and include anendoscope, a display apparatus which displays an image picked up by theendoscope and so forth. Various apparatus belonging to the endoscopicsurgery system 5113 are referred to also as medical equipment.Meanwhile, the display apparatus 5103A to 5103D, the recorder 5105, thepatient bed 5183 and the illumination 5191 are apparatus which areequipped, for example, in the surgery room separately from theendoscopic surgery system 5113. The apparatus which do not belong to theendoscopic surgery system 5113 are referred to also as non-medicalequipment. The audiovisual controller 5107 and/or the surgery roomcontrolling apparatus 5109 cooperatively control operation of themedical equipment and the non-medical equipment with each other.

The audiovisual controller 5107 integrally controls processes of themedical equipment and the non-medical equipment relating to imagedisplay. Specifically, each of the apparatus group 5101, the ceilingcamera 5187 and the surgery field camera 5189 from among the apparatusprovided in the surgery room system 5100 may be an apparatus having afunction of sending information to be displayed during surgery (suchinformation is hereinafter referred to as display information, and theapparatus mentioned is hereinafter referred to as apparatus of a sendingsource). Meanwhile, each of the display apparatus 5103A to 5103D may bean apparatus to which display information is outputted (the apparatus ishereinafter referred to also as apparatus of an output destination).Further, the recorder 5105 may be an apparatus which serves as both ofan apparatus of a sending source and an apparatus of an outputdestination. The audiovisual controller 5107 has a function ofcontrolling operation of an apparatus of a sending source and anapparatus of an output destination to acquire display information fromthe apparatus of a sending source and transmit the display informationto the apparatus of an output destination so as to be displayed orrecorded. It is to be noted that the display information includesvarious images picked up during surgery, various kinds of informationrelating to the surgery (for example, physical information of a patient,inspection results in the past or information regarding a surgicalprocedure) and so forth.

Specifically, to the audiovisual controller 5107, information relatingto an image of a surgical region in a body lumen of a patient imaged bythe endoscope may be transmitted as the display information from theapparatus group 5101. Further, from the ceiling camera 5187, informationrelating to an image of the hands of the surgeon picked up by theceiling camera 5187 may be transmitted as display information. Further,from the surgery field camera 5189, information relating to an imagepicked up by the surgery field camera 5189 and illustrating a state ofthe entire surgery room may be transmitted as display information. It isto be noted that, if a different apparatus having an image pickupfunction exists in the surgery room system 5100, then the audiovisualcontroller 5107 may acquire information relating to an image picked upby the different apparatus as display information also from thedifferent apparatus.

Alternatively, for example, in the recorder 5105, information relatingto such images as mentioned above picked up in the past is recorded bythe audiovisual controller 5107. The audiovisual controller 5107 canacquire, as display information, information relating to the imagespicked up in the past from the recorder 5105. It is to be noted thatalso various pieces of information relating to surgery may be recordedin advance in the recorder 5105.

The audiovisual controller 5107 controls at least one of the displayapparatus 5103A to 5103D, which are apparatus of an output destination,to display acquired display information (namely, images picked up duringsurgery or various pieces of information relating to the surgery). Inthe example depicted, the display apparatus 5103A is a display apparatusinstalled so as to be suspended from the ceiling of the surgery room;the display apparatus 5103B is a display apparatus installed on a wallface of the surgery room; the display apparatus 5103C is a displayapparatus installed on a desk in the surgery room; and the displayapparatus 5103D is a mobile apparatus (for example, a tablet personalcomputer (PC)) having a display function.

Further, though not depicted in FIG. 36, the surgery room system 5100may include an apparatus outside the surgery room. The apparatus outsidethe surgery room may be, for example, a server connected to a networkconstructed inside and outside the hospital, a PC used by medical staff,a projector installed in a meeting room of the hospital or the like.Where such an external apparatus is located outside the hospital, alsoit is possible for the audiovisual controller 5107 to cause displayinformation to be displayed on a display apparatus of a differenthospital through a teleconferencing system or the like to performtelemedicine.

The surgery room controlling apparatus 5109 integrally controlsprocesses other than processes relating to image display on thenon-medical equipment. For example, the surgery room controllingapparatus 5109 controls driving of the patient bed 5183, the ceilingcamera 5187, the surgery field camera 5189 and the illumination 5191.

In the surgery room system 5100, a centralized operation panel 5111 isprovided such that it is possible to issue an instruction regardingimage display to the audiovisual controller 5107 or issue an instructionregarding operation of the non-medical equipment to the surgery roomcontrolling apparatus 5109 through the centralized operation panel 5111.The centralized operation panel 5111 is configured by providing a touchpanel on a display face of a display apparatus.

FIG. 37 is a view depicting an example of display of an operation screenimage on the centralized operation panel 5111. In FIG. 37, as anexample, an operation screen image is depicted which corresponds to acase in which two display apparatus are provided as apparatus of anoutput destination in the surgery room system 5100. Referring to FIG.37, the operation screen image 5193 includes a sending source selectionregion 5195, a preview region 5197 and a control region 5201.

In the sending source selection region 5195, the sending sourceapparatus provided in the surgery room system 5100 and thumbnail screenimages representative of display information the sending sourceapparatus have are displayed in an associated manner with each other. Auser can select display information to be displayed on the displayapparatus from any of the sending source apparatus displayed in thesending source selection region 5195.

In the preview region 5197, a preview of screen images displayed on twodisplay apparatus (Monitor 1 and Monitor 2) which are apparatus of anoutput destination is displayed. In the example depicted, four imagesare displayed by picture in picture (PinP) display in regard to onedisplay apparatus. The four images correspond to display informationsent from the sending source apparatus selected in the sending sourceselection region 5195. One of the four images is displayed in acomparatively large size as a main image while the remaining threeimages are displayed in a comparatively small size as sub images. Theuser can exchange between the main image and the sub images by suitablyselecting one of the images from among the four images displayed in theregion. Further, a status displaying region 5199 is provided below theregion in which the four images are displayed, and a status relating tosurgery (for example, elapsed time of the surgery, physical informationof the patient and so forth) may be displayed suitably in the statusdisplaying region 5199.

A sending source operation region 5203 and an output destinationoperation region 5205 are provided in the control region 5201. In thesending source operation region 5203, a graphical user interface (GUI)part for performing an operation for an apparatus of a sending source isdisplayed. In the output destination operation region 5205, a GUI partfor performing an operation for an apparatus of an output destination isdisplayed. In the example depicted, GUI parts for performing variousoperations for a camera (panning, tilting and zooming) in an apparatusof a sending source having an image pickup function are provided in thesending source operation region 5203. The user can control operation ofthe camera of an apparatus of a sending source by suitably selecting anyof the GUI parts. It is to be noted that, though not depicted, where theapparatus of a sending source selected in the sending source selectionregion 5195 is a recorder (namely, where an image recorded in therecorder in the past is displayed in the preview region 5197), GUI partsfor performing such operations as reproduction of the image, stopping ofreproduction, rewinding, fast-feeding and so forth may be provided inthe sending source operation region 5203.

Further, in the output destination operation region 5205, GUI parts forperforming various operations for display on a display apparatus whichis an apparatus of an output destination (swap, flip, color adjustment,contrast adjustment and switching between two dimensional (2D) displayand three dimensional (3D) display) are provided. The user can operatethe display of the display apparatus by suitably selecting any of theGUI parts.

It is to be noted that the operation screen image to be displayed on thecentralized operation panel 5111 is not limited to the depicted example,and the user may be able to perform operation inputting to eachapparatus which can be controlled by the audiovisual controller 5107 andthe surgery room controlling apparatus 5109 provided in the surgery roomsystem 5100 through the centralized operation panel 5111.

FIG. 38 is a view illustrating an example of a state of surgery to whichthe surgery room system described above is applied. The ceiling camera5187 and the surgery field camera 5189 are provided on the ceiling ofthe surgery room such that it can image the hands of a surgeon (medicaldoctor) 5181 who performs treatment for an affected area of a patient5185 on the patient bed 5183 and the entire surgery room. The ceilingcamera 5187 and the surgery field camera 5189 may include amagnification adjustment function, a focal distance adjustment function,an imaging direction adjustment function and so forth. The illumination5191 is provided on the ceiling of the surgery room and irradiates atleast upon the hands of the surgeon 5181. The illumination 5191 may beconfigured such that the irradiation light amount, the wavelength(color) of the irradiation light, the irradiation direction of the lightand so forth can be adjusted suitably.

The endoscopic surgery system 5113, the patient bed 5183, the ceilingcamera 5187, the surgery field camera 5189 and the illumination 5191 areconnected for cooperation with each other through the audiovisualcontroller 5107 and the surgery room controlling apparatus 5109 (notdepicted in FIG. 38) as depicted in FIG. 36. The centralized operationpanel 5111 is provided in the surgery room, and the user can suitablyoperate the apparatus existing in the surgery room through thecentralized operation panel 5111 as described hereinabove.

In the following, a configuration of the endoscopic surgery system 5113is described in detail. As depicted, the endoscopic surgery system 5113includes an endoscope 5115, other surgical tools 5131, a supporting armapparatus 5141 which supports the endoscope 5115 thereon, and a cart5151 on which various apparatus for endoscopic surgery are mounted.

In endoscopic surgery, in place of incision of the abdominal wall toperform laparotomy, a plurality of tubular aperture devices calledtrocars 5139 a to 5139 d are used to puncture the abdominal wall. Then,a lens barrel 5117 of the endoscope 5115 and the other surgical tools5131 are inserted into body lumens of the patient 5185 through thetrocars 5139 a to 5139 d. In the example depicted, as the other surgicaltools 5131, a pneumoperitoneum tube 5133, an energy treatment tool 5135and forceps 5137 are inserted into body lumens of the patient 5185.Further, the energy treatment tool 5135 is a treatment tool forperforming incision and peeling of a tissue, sealing of a blood vesselor the like by high frequency current or ultrasonic vibration. However,the surgical tools 5131 depicted are mere examples at all, and as thesurgical tools 5131, various surgical tools which are generally used inendoscopic surgery such as, for example, a pair of tweezers or aretractor may be used.

An image of a surgical region in a body lumen of the patient 5185 pickedup by the endoscope 5115 is displayed on a display apparatus 5155. Thesurgeon 5181 would use the energy treatment tool 5135 or the forceps5137 while watching the image of the surgical region displayed on thedisplay apparatus 5155 on the real time basis to perform such treatmentas, for example, resection of an affected area. It is to be noted that,though not depicted, the pneumoperitoneum tube 5133, the energytreatment tool 5135, and the forceps 5137 are supported by the surgeon5181, an assistant or the like during surgery.

(Supporting Arm Apparatus)

The supporting arm apparatus 5141 includes an arm unit 5145 extendingfrom a base unit 5143. In the example depicted, the arm unit 5145includes joint portions 5147 a, 5147 b and 5147 c and links 5149 a and5149 b and is driven under the control of an arm controlling apparatus5159. The endoscope 5115 is supported by the arm unit 5145 such that theposition and the posture of the endoscope 5115 are controlled.Consequently, stable fixation in position of the endoscope 5115 can beimplemented.

(Endoscope)

The endoscope 5115 includes the lens barrel 5117 which has a region of apredetermined length from a distal end thereof to be inserted into abody lumen of the patient 5185, and a camera head 5119 connected to aproximal end of the lens barrel 5117. In the example depicted, theendoscope 5115 is depicted which is configured as a hard mirror havingthe lens barrel 5117 of the hard type. However, the endoscope 5115 mayotherwise be configured as a soft mirror having the lens barrel 5117 ofthe soft type.

The lens barrel 5117 has, at a distal end thereof, an opening in whichan objective lens is fitted. A light source apparatus 5157 is connectedto the endoscope 5115 such that light generated by the light sourceapparatus 5157 is introduced to a distal end of the lens barrel 5117 bya light guide extending in the inside of the lens barrel 5117 and isapplied toward an observation target in a body lumen of the patient 5185through the objective lens. It is to be noted that the endoscope 5115may be a direct view mirror or may be a perspective view mirror or aside view mirror.

An optical system and an image pickup element are provided in the insideof the camera head 5119 such that reflected light (observation light)from an observation target is condensed on the image pickup element bythe optical system. The observation light is photo-electricallyconverted by the image pickup element to generate an electric signalcorresponding to the observation light, namely, an image signalcorresponding to an observation image. The image signal is transmittedas RAW data to a CCU 5153. It is to be noted that the camera head 5119has a function incorporated therein for suitably driving the opticalsystem of the camera head 5119 to adjust the magnification and the focaldistance.

It is to be noted that, in order to establish compatibility with, forexample, a stereoscopic vision (3D display), a plurality of image pickupelements may be provided on the camera head 5119. In this case, aplurality of relay optical systems are provided in the inside of thelens barrel 5117 in order to guide observation light to the plurality ofrespective image pickup elements.

(Various Apparatus Incorporated in Cart)

The CCU 5153 includes a central processing unit (CPU), a graphicsprocessing unit (GPU) or the like and integrally controls operation ofthe endoscope 5115 and the display apparatus 5155. Specifically, the CCU5153 performs, for an image signal received from the camera head 5119,various image processes for displaying an image based on the imagesignal such as, for example, a development process (demosaic process).The CCU 5153 provides the image signal for which the image processeshave been performed to the display apparatus 5155. Further, theaudiovisual controller 5107 depicted in FIG. 36 is connected to the CCU5153. The CCU 5153 provides the image signal for which the imageprocesses have been performed also to the audiovisual controller 5107.Further, the CCU 5153 transmits a control signal to the camera head 5119to control driving of the camera head 5119. The control signal mayinclude information relating to an image pickup condition such as amagnification or a focal distance. The information relating to an imagepickup condition may be inputted through the inputting apparatus 5161 ormay be inputted through the centralized operation panel 5111 describedhereinabove.

The display apparatus 5155 displays an image based on an image signalfor which the image processes have been performed by the CCU 5153 underthe control of the CCU 5153. If the endoscope 5115 is ready for imagingof a high resolution such as 4K (horizontal pixel number 3840×verticalpixel number 2160), 8K (horizontal pixel number 7680×vertical pixelnumber 4320) or the like and/or ready for 3D display, then a displayapparatus by which corresponding display of the high resolution and/or3D display are possible may be used as the display apparatus 5155. Wherethe apparatus is ready for imaging of a high resolution such as 4K or8K, if the display apparatus used as the display apparatus 5155 has asize of equal to or not less than 55 inches, then a more immersiveexperience can be obtained. Further, a plurality of display apparatus5155 having different resolutions and/or different sizes may be providedin accordance with purposes.

The light source apparatus 5157 includes a light source such as, forexample, a light emitting diode (LED) and supplies irradiation light forimaging of a surgical region to the endoscope 5115.

The arm controlling apparatus 5159 includes a processor such as, forexample, a CPU and operates in accordance with a predetermined programto control driving of the arm unit 5145 of the supporting arm apparatus5141 in accordance with a predetermined controlling method.

An inputting apparatus 5161 is an input interface for the endoscopicsurgery system 5113. A user can perform inputting of various kinds ofinformation or instruction inputting to the endoscopic surgery system5113 through the inputting apparatus 5161. For example, the user wouldinput various kinds of information relating to surgery such as physicalinformation of a patient, information regarding a surgical procedure ofthe surgery and so forth through the inputting apparatus 5161. Further,the user would input, for example, an instruction to drive the arm unit5145, an instruction to change an image pickup condition (type ofirradiation light, magnification, focal distance or the like) by theendoscope 5115, an instruction to drive the energy treatment tool 5135or a like through the inputting apparatus 5161.

The type of the inputting apparatus 5161 is not limited and may be thatof any one of various known inputting apparatus. As the inputtingapparatus 5161, for example, a mouse, a keyboard, a touch panel, aswitch, a foot switch 5171 and/or a lever or the like may be applied.Where a touch panel is used as the inputting apparatus 5161, it may beprovided on the display face of the display apparatus 5155.

The inputting apparatus 5161 is otherwise a device to be mounted on auser such as, for example, a glasses type wearable device or a headmounted display (HMD), and various kinds of inputting are performed inresponse to a gesture or a line of sight of the user detected by any ofthe devices mentioned. Further, the inputting apparatus 5161 includes acamera which can detect a motion of a user, and various kinds ofinputting are performed in response to a gesture or a line of sight of auser detected from a video picked up by the camera. Further, theinputting apparatus 5161 includes a microphone which can collect thevoice of a user, and various kinds of inputting are performed by voicethrough the microphone. By configuring the inputting apparatus 5161 suchthat various kinds of information can be inputted in a contactlessfashion in this manner, especially a user who belongs to a clean area(for example, the surgeon 5181) can operate an apparatus belonging to anunclean area in a contactless fashion. Further, since the user canoperate an apparatus without releasing a possessed surgical tool fromits hand, the convenience to the user is improved.

A treatment tool controlling apparatus 5163 controls driving of theenergy treatment tool 5135 for cautery or incision of a tissue, sealingof a blood vessel or the like. A pneumoperitoneum apparatus 5165 feedsgas into a body lumen of the patient 5185 through the pneumoperitoneumtube 5133 to inflate the body lumen in order to secure the field of viewof the endoscope 5115 and secure the working space for the surgeon. Arecorder 5167 is an apparatus capable of recording various kinds ofinformation relating to surgery. A printer 5169 is an apparatus capableof printing various kinds of information relating to surgery in variousforms such as a text, an image or a graph.

In the following, especially a characteristic configuration of theendoscopic surgery system 5113 is described in more detail.

(Supporting Arm Apparatus)

The supporting arm apparatus 5141 includes the base unit 5143 serving asa base, and the arm unit 5145 extending from the base unit 5143. In theexample depicted, the arm unit 5145 includes the plurality of jointportions 5147 a, 5147 b and 5147 c and the plurality of links 5149 a and5149 b connected to each other by the joint portion 5147 b. In FIG. 38,for simplified illustration, the configuration of the arm unit 5145 isdepicted in a simplified form. Actually, the shape, number andarrangement of the joint portions 5147 a to 5147 c and the links 5149 aand 5149 b and the direction and so forth of axes of rotation of thejoint portions 5147 a to 5147 c can be set suitably such that the armunit 5145 has a desired degree of freedom. For example, the arm unit5145 may preferably be included such that it has a degree of freedomequal to or not less than 6 degrees of freedom. This makes it possibleto move the endoscope 5115 freely within the movable range of the armunit 5145. Consequently, it becomes possible to insert the lens barrel5117 of the endoscope 5115 from a desired direction into a body lumen ofthe patient 5185.

An actuator is provided in the joint portions 5147 a to 5147 c, and thejoint portions 5147 a to 5147 c include such that they are rotatablearound predetermined axes of rotation thereof by driving of theactuator. The driving of the actuator is controlled by the armcontrolling apparatus 5159 to control the rotational angle of each ofthe joint portions 5147 a to 5147 c thereby to control driving of thearm unit 5145. Consequently, control of the position and the posture ofthe endoscope 5115 can be implemented. Thereupon, the arm controllingapparatus 5159 can control driving of the arm unit 5145 by various knowncontrolling methods such as force control or position control.

For example, if the surgeon 5181 suitably performs operation inputtingthrough the inputting apparatus 5161 (including the foot switch 5171),then driving of the arm unit 5145 may be controlled suitably by the armcontrolling apparatus 5159 in response to the operation input to controlthe position and the posture of the endoscope 5115. After the endoscope5115 at the distal end of the arm unit 5145 is moved from an arbitraryposition to a different arbitrary position by the control justdescribed, the endoscope 5115 can be supported fixedly at the positionafter the movement. It is to be noted that the arm unit 5145 may beoperated in a master-slave fashion. In this case, the arm unit 5145 maybe remotely controlled by the user through the inputting apparatus 5161which is placed at a place remote from the surgery room.

Further, where force control is applied, the arm controlling apparatus5159 may perform power-assisted control to drive the actuators of thejoint portions 5147 a to 5147 c such that the arm unit 5145 may receiveexternal force by the user and move smoothly following the externalforce. This makes it possible to move the arm unit 5145 withcomparatively weak force when the user directly touches with and movesthe arm unit 5145. Accordingly, it becomes possible for the user to movethe endoscope 5115 more intuitively by a simpler and easier operation,and the convenience to the user can be improved.

Here, generally in endoscopic surgery, the endoscope 5115 is supportedby a medical doctor called scopist. In contrast, where the supportingarm apparatus 5141 is used, the position of the endoscope 5115 can befixed with a higher degree of certainty without hands, and therefore, animage of a surgical region can be obtained stably and surgery can beperformed smoothly.

It is to be noted that the arm controlling apparatus 5159 may notnecessarily be provided on the cart 5151. Further, the arm controllingapparatus 5159 may not necessarily be a single apparatus. For example,the arm controlling apparatus 5159 may be provided in each of the jointportions 5147 a to 5147 c of the arm unit 5145 of the supporting armapparatus 5141 such that the plurality of arm controlling apparatus 5159cooperate with each other to implement driving control of the arm unit5145.

(Light Source Apparatus)

The light source apparatus 5157 supplies irradiation light upon imagingof a surgical region to the endoscope 5115. The light source apparatus5157 includes a white light source which includes, for example, an LED,a laser light source or a combination of them. In this case, where awhite light source includes a combination of red, green, and blue (RGB)laser light sources, since the output intensity and the output timingcan be controlled with a high degree of accuracy for each color (eachwavelength), adjustment of the white balance of a picked up image can beperformed by the light source apparatus 5157. Further, in this case, iflaser beams from the RGB laser light sources are appliedtime-divisionally on an observation target and driving of the imagepickup elements of the camera head 5119 is controlled in synchronismwith the irradiation timings, then images individually corresponding tothe R, G and B colors can be picked up time-divisionally. According tothe method just described, a color image can be obtained even if a colorfilter is not provided for the image pickup element.

Further, driving of the light source apparatus 5157 may be controlledsuch that the intensity of light to be outputted is changed for eachpredetermined time. By controlling driving of the image pickup elementof the camera head 5119 in synchronism with the timing of the change ofthe intensity of light to acquire images time-divisionally andsynthesizing the images, an image of a high dynamic range free fromunderexposed blocked up shadows and overexposed highlights can becreated.

Further, the light source apparatus 5157 may be configured to supplylight of a predetermined wavelength band ready for special lightobservation. In special light observation, for example, by utilizing thewavelength dependency of absorption of light of a body tissue, narrowband light observation (narrow band imaging) of imaging a predeterminedtissue such as a blood vessel of a superficial portion of the mucousmembrane or the like in a high contrast is performed by applying lightof a narrower band in comparison with irradiation light upon ordinaryobservation (namely, white light). Alternatively, in special lightobservation, fluorescent observation for obtaining an image fromfluorescent light generated by irradiation of excitation light may alsobe performed. In fluorescent observation, it is possible to performobservation of fluorescent light from a body tissue by irradiatingexcitation light on the body tissue (autofluorescence observation) or toobtain a fluorescent light image by locally injecting a reagent such asindocyanine green (ICG) into a body tissue and irradiating excitationlight corresponding to a fluorescent light wavelength of the reagentupon the body tissue. The light source apparatus 5157 can be configuredto supply such narrow-band light and/or excitation light suitable forspecial light observation as described above.

(Camera Head and CCU)

Functions of the camera head 5119 of the endoscope 5115 and the CCU 5153are described in more detail with reference to FIG. 39. FIG. 39 is ablock diagram depicting an example of a functional configuration of thecamera head 5119 and the CCU 5153 depicted in FIG. 38.

Referring to FIG. 39, the camera head 5119 has, as functions thereof, alens unit 5121, an image pickup unit 5123, a driving unit 5125, acommunication unit 5127 and a camera head controlling unit 5129.Further, the CCU 5153 has, as functions thereof, a communication unit5173, an image processing unit 5175 and a control unit 5177. The camerahead 5119 and the CCU 5153 are connected to be bidirectionallycommunicable to each other by a transmission cable 5179.

First, a functional configuration of the camera head 5119 is described.The lens unit 5121 is an optical system provided at a connectinglocation of the camera head 5119 to the lens barrel 5117. Observationlight taken in from a distal end of the lens barrel 5117 is introducedinto the camera head 5119 and enters the lens unit 5121. The lens unit5121 includes a combination of a plurality of lenses including a zoomlens and a focusing lens. The lens unit 5121 has optical propertiesadjusted such that the observation light is condensed on a lightreceiving face of the image pickup element of the image pickup unit5123. Further, the zoom lens and the focusing lens include such that thepositions thereof on their optical axis are movable for adjustment ofthe magnification and the focal point of a picked up image.

The image pickup unit 5123 includes an image pickup element and disposedat a succeeding stage to the lens unit 5121. Observation light havingpassed through the lens unit 5121 is condensed on the light receivingface of the image pickup element, and an image signal corresponding tothe observation image is generated by photoelectric conversion. Theimage signal generated by the image pickup unit 5123 is provided to thecommunication unit 5127.

As the image pickup element which is included by the image pickup unit5123, an image sensor, for example, of the complementary metal oxidesemiconductor (CMOS) type is used which has a Bayer array and is capableof picking up an image in color. It is to be noted that, as the imagepickup element, an image pickup element may be used which is ready, forexample, for imaging of an image of a high resolution equal to or notless than 4K. If an image of a surgical region is obtained in a highresolution, then the surgeon 5181 can comprehend a state of the surgicalregion in enhanced details and can proceed with the surgery moresmoothly.

Further, the image pickup element which is included by the image pickupunit 5123 is configured such that it has a pair of image pickup elementsfor acquiring image signals for the right eye and the left eyecompatible with 3D display. Where 3D display is applied, the surgeon5181 can comprehend the depth of a living body tissue in the surgicalregion with a higher degree of accuracy. It is to be noted that, if theimage pickup unit 5123 is configured as that of the multi-plate type,then a plurality of systems of lens units 5121 are providedcorresponding to the individual image pickup elements of the imagepickup unit 5123.

The image pickup unit 5123 may not necessarily be provided on the camerahead 5119. For example, the image pickup unit 5123 may be provided justbehind the objective lens in the inside of the lens barrel 5117.

The driving unit 5125 includes an actuator and moves the zoom lens andthe focusing lens of the lens unit 5121 by a predetermined distancealong the optical axis under the control of the camera head controllingunit 5129. Consequently, the magnification and the focal point of apicked up image by the image pickup unit 5123 can be adjusted suitably.

The communication unit 5127 includes a communication apparatus fortransmitting and receiving various kinds of information to and from theCCU 5153. The communication unit 5127 transmits an image signal acquiredfrom the image pickup unit 5123 as RAW data to the CCU 5153 through thetransmission cable 5179. Thereupon, in order to display a picked upimage of a surgical region in low latency, preferably the image signalis transmitted by optical communication. This is because, since, uponsurgery, the surgeon 5181 performs surgery while observing the state ofan affected area through a picked up image, in order to achieve surgerywith a higher degree of safety and certainty, it is demanded for amoving image of the surgical region to be displayed on the real timebasis as far as possible. Where optical communication is applied, aphotoelectric conversion module for converting an electric signal intoan optical signal is provided in the communication unit 5127. After theimage signal is converted into an optical signal by the photoelectricconversion module, it is transmitted to the CCU 5153 through thetransmission cable 5179.

Further, the communication unit 5127 receives a control signal forcontrolling driving of the camera head 5119 from the CCU 5153. Thecontrol signal includes information relating to image pickup conditionssuch as, for example, information that a frame rate of a picked up imageis designated, information that an exposure value upon image picking upis designated and/or information that a magnification and a focal pointof a picked up image are designated. The communication unit 5127provides the received control signal to the camera head controlling unit5129. It is to be noted that also the control signal from the CCU 5153may be transmitted by optical communication. In this case, aphotoelectric conversion module for converting an optical signal into anelectric signal is provided in the communication unit 5127. After thecontrol signal is converted into an electric signal by the photoelectricconversion module, it is provided to the camera head controlling unit5129.

It is to be noted that the image pickup conditions such as the framerate, exposure value, magnification or focal point are set automaticallyby the control unit 5177 of the CCU 5153 on the basis of an acquiredimage signal. In other words, an auto exposure (AE) function, an autofocus (AF) function and an auto white balance (AWB) function areincorporated in the endoscope 5115.

The camera head controlling unit 5129 controls driving of the camerahead 5119 on the basis of a control signal from the CCU 5153 receivedthrough the communication unit 5127. For example, the camera headcontrolling unit 5129 controls driving of the image pickup element ofthe image pickup unit 5123 on the basis of information that a frame rateof a picked up image is designated and/or information that an exposurevalue upon image picking up is designated. Further, for example, thecamera head controlling unit 5129 controls the driving unit 5125 tosuitably move the zoom lens and the focus lens of the lens unit 5121 onthe basis of information that a magnification and a focal point of apicked up image are designated. The camera head controlling unit 5129may include a function for storing information for identifying of thelens barrel 5117 and/or the camera head 5119.

It is to be noted that, by disposing the components such as the lensunit 5121 and the image pickup unit 5123 in a sealed structure havinghigh airtightness and high waterproof, the camera head 5119 can beprovided with resistance to an autoclave sterilization process.

Now, a functional configuration of the CCU 5153 is described. Thecommunication unit 5173 includes a communication apparatus fortransmitting and receiving various kinds of information to and from thecamera head 5119. The communication unit 5173 receives an image signaltransmitted thereto from the camera head 5119 through the transmissioncable 5179. Thereupon, the image signal may be transmitted preferably byoptical communication as described above. In this case, for thecompatibility with optical communication, the communication unit 5173includes a photoelectric conversion module for converting an opticalsignal into an electric signal. The communication unit 5173 provides theimage signal after conversion into an electric signal to the imageprocessing unit 5175.

Further, the communication unit 5173 transmits, to the camera head 5119,a control signal for controlling driving of the camera head 5119. Alsothe control signal may be transmitted by optical communication.

The image processing unit 5175 performs various image processes for animage signal in the form of RAW data transmitted thereto from the camerahead 5119. The image processes include various known signal processessuch as, for example, a development process, an image quality improvingprocess (a bandwidth enhancement process, a super-resolution process, anoise reduction (NR) process and/or an image stabilization process)and/or an enlargement process (electronic zooming process). Further, theimage processing unit 5175 performs a detection process for an imagesignal for performing AE, AF and AWB.

The image processing unit 5175 includes a processor such as a CPU or aGPU, and when the processor operates in accordance with a predeterminedprogram, the image processes and the detection process described abovecan be performed. It is to be noted that, where the image processingunit 5175 includes a plurality of GPUs, the image processing unit 5175suitably divides information relating to an image signal such that imageprocesses are performed in parallel by the plurality of GPUs.

The control unit 5177 performs various kinds of control relating toimage picking up of a surgical region by the endoscope 5115 and displayof the picked up image. For example, the control unit 5177 generates acontrol signal for controlling driving of the camera head 5119.Thereupon, if image pickup conditions are inputted by the user, then thecontrol unit 5177 generates a control signal on the basis of the inputby the user. Alternatively, where the endoscope 5115 has an AE function,an AF function and an AWB function incorporated therein, the controlunit 5177 suitably calculates an optimum exposure value, focal distanceand white balance in response to a result of a detection process by theimage processing unit 5175 and generates a control signal.

Further, the control unit 5177 controls the display apparatus 5155 todisplay an image of a surgical region on the basis of an image signalfor which the image processes have been performed by the imageprocessing unit 5175. Thereupon, the control unit 5177 recognizesvarious objects in the surgical region image using various imagerecognition technologies. For example, the control unit 5177 canrecognize a surgical tool such as forceps, a particular living bodyregion, bleeding, mist when the energy treatment tool 5135 is used andso forth by detecting the shape, color and so forth of edges of theobjects included in the surgical region image. The control unit 5177causes, when it controls the display apparatus 5155 to display asurgical region image, various kinds of surgery supporting informationto be displayed in an overlapping manner with an image of the surgicalregion using a result of the recognition. Where surgery supportinginformation is displayed in an overlapping manner and presented to thesurgeon 5181, the surgeon 5181 can proceed with the surgery more safetyand certainty.

The transmission cable 5179 which connects the camera head 5119 and theCCU 5153 to each other is an electric signal cable ready forcommunication of an electric signal, an optical fiber ready for opticalcommunication or a composite cable thereof.

Here, while, in the example depicted in the figure, communication isperformed by wired communication using the transmission cable 5179, thecommunication between the camera head 5119 and the CCU 5153 may beperformed otherwise by wireless communication. Where the communicationbetween the camera head 5119 and the CCU 5153 is performed by wirelesscommunication, there is no necessity to lay the transmission cable 5179in the surgery room. Therefore, such a situation that movement ofmedical staff in the surgery room is disturbed by the transmission cable5179 can be eliminated.

An example of the surgery room system 5100 to which the technologyaccording to an embodiment of the present disclosure can be applied hasbeen described above. It is to be noted here that, although a case inwhich the medical system to which the surgery room system 5100 isapplied is the endoscopic surgery system 5113 has been described as anexample, the configuration of the surgery room system 5100 is notlimited to that of the example described above. For example, the surgeryroom system 5100 may be applied to a soft endoscopic system forinspection or a microscopic surgery system in place of the endoscopicsurgery system 5113.

The technology according to the present disclosure is applicable to theaudiovisual controller 5107 in the configuration described above. Morespecifically, for example, the two ceiling cameras 5187 are disposed insuch positions as to image in different directions, and the audiovisualcontroller 5107 corresponding to the control unit of the presenttechnology is configured to control the imaging directions of the twoceiling cameras 5187.

The audiovisual controller 5107 controls the imaging directions of thetwo ceiling cameras 5187 in accordance with a situation of an objectassociated with each of a first image captured by the first ceilingcamera 5187, and a second image captured by the second ceiling camera5187.

By applying the technology according to the present disclosure to theaudiovisual controller 5107, two images of the hands of the surgeoncaptured in different directions by the two ceiling cameras 5187 areoutputted while appropriately switched to each other in accordance withthe standing position of the surgeon, the positions of the hands of thesurgeon, and the position of the surgical region, for example.Accordingly, the state of the surgery can be recorded without omissionsand invisible areas of the surgical region behind the body, the hands orthe like of the surgeon.

Note that embodiments according to the present technology are notlimited to the embodiments described above, but may be modified invarious manners without departing from the subject matters of thepresent technology.

Moreover, advantageous effects to be offered are not limited to theadvantageous effects described in the present description and given onlyby way of example, but may include other advantageous effects.

Furthermore, the present technology may have following configurations.

-   (1)

A controller including:

a control unit that controls a first imaging direction of a firstimaging unit and a second imaging direction of a second imaging unit asa direction different from the first imaging direction in accordancewith a situation of an object associated with a first image captured bythe first imaging unit or a second image captured by the second imagingunit.

-   (2)

The controller according to (1), in which the control unit controls thefirst imaging direction and the second imaging direction in accordancewith a behavior of the object of the first image or the second image.

-   (3)

The controller according to (2), in which the behavior of the objectincludes an action of the object.

-   (4)

The controller according to (3), in which the action of the objectincludes at least any one of standing up, sitting down, raising hand, ormoving.

-   (5)

The controller according to (2), in which the behavior of the objectincludes a sound emitted from the object.

-   (6)

The controller according to (5), in which the sound emitted from theobject includes a spoken voice.

-   (7)

The controller according to any one of (1) to (6), in which

the control unit executes moving body detection, gesture detection, andtracking of the object for the first image,

the control unit executes moving body detection, gesture detection, andtracking of the object for the second image, and

the control unit controls the first imaging direction and the secondimaging direction in accordance with a result of any one of the movingbody detection, the gesture detection, and the tracking for the firstimage or the second image.

-   (8)

The controller according to (7), in which the control unit controlsoutput of at least either the first image or the second image inaccordance with the situation of the object associated with the firstimage or the second image.

-   (9)

The controller according to (8), in which the control unit outputs thefirst image or the second image while switching between the first imageand the second image in accordance with the situation of the object ofthe first image or the second image.

-   (10)

The controller according to (8), in which the control unit merges thefirst image and the second image side by side and outputs the mergedimage in accordance with the situation of the object of the first imageor the second image.

-   (11)

The controller according to (8), in which the control unit merges thefirst image and the second image in a picture-in-picture manner andoutputs the merged image in accordance with the situation of the objectof the first image or the second image.

-   (12)

The controller according to any one of (8) to (11), in which

the first image is acquired by imaging a lecturer located in the firstimaging direction,

the second image is acquired by imaging an auditor located in the secondimaging direction,

the control unit performs a board extraction process for a third imageacquired by imaging a blackboard or whiteboard located in a thirdimaging direction different from the first imaging direction and thesecond imaging direction to extract contents of board writing, and

the control unit controls the first to third imaging directions inaccordance with a result of any one of the moving body detection, thegesture detection, and the tracking for the first image or the secondimage and the board extraction process, and controls output of any oneof the first image, the second image, and the contents of the boardwriting.

-   (13)

The controller according to any one of (8) to (11), in which

the first image is acquired by imaging a lecturer located in the firstimaging direction,

the second image is acquired by imaging an auditor located in the secondimaging direction, and

the control unit controls the first and second imaging directions inaccordance with a result of any one of the moving body detection, thegesture detection, and the tracking for the first image or the secondimage or an output change of a third image outputted from an informationprocessing apparatus, and controls output of any one of the first image,the second image, or the third image.

-   (14)

The controller according to any one of (1) to (13), in which

the control unit controls actions of respective control targets on abasis of a scenario that indicates a shift of a state of each of thecontrol targets,

the scenario describes a Behavior and an Event for each of a pluralityof the states, and

the Behavior describes actions executed by the respective controltargets, and the Event describes a shift condition for shifting to adifferent one of the states, and a shift destination of the stateassociated with the shift condition.

-   (15)

The controller according to (14), in which the state shifts to a secondstate associated with the shift condition in a case where any one of thecontrol targets meets the shift condition described in the Event of afirst state during execution of actions described in the Behavior of thefirst state by the respective control targets in the first state on thebasis of the scenario.

-   (16)

The controller according to (15), in which the shift condition includesa condition based on a change of the situation of the object in thefirst image or the second image.

-   (17)

The controller according to any one of (14) to (16), in which thescenario is expressed by a state shift chart or a timeline.

-   (18)

The controller according to any one of (14) to (17), in which thecontrol unit adds and deletes the states in the scenario and changesdescription contents of the states on a basis of an operation by a user.

-   (19)

The controller according to (18), in which the control unit controlsdisplay of GUI that receives addition and deletion of the states in thescenario and a change of the description contents of the states.

-   (20)

A control method of a controller, including:

controlling a first imaging direction of a first imaging unit and asecond imaging direction of a second imaging unit as a directiondifferent from the first imaging direction in accordance with asituation of an object associated with a first image captured by thefirst imaging unit or a second image captured by the second imagingunit.

-   (21)

A controller including:

a control unit that controls output of at least either a first imageacquired by imaging in a first imaging direction or a second imagedifferent from the first image in accordance with at least a situationof an object of the first image.

-   (22)

The controller according to (21), in which

the control unit acquires the second image by controlling imaging in asecond imaging direction different from the first imaging direction, and

the control unit controls output of at least either the first image orthe second image in accordance with a situation of an object of thefirst image or the second image.

-   (23)

The controller according to (21), in which

the first image is acquired by imaging a person located in the firstimaging direction,

the control unit executes moving body detection, gesture detection, andtracking of the object for the first image,

the control unit acquires a result of a board extraction process thatextracts contents of board writing from the second image acquired byimaging a blackboard or whiteboard located in a second imaging directiondifferent from the first imaging direction, and

the control unit controls output of at least either the first image orthe contents of the board writing in accordance with a result of themoving body detection, the gesture detection, and the tracking for thefirst image or the board extraction process.

-   (24)

The controller according to (21), in which

the first image is acquired by imaging a person located in the firstimaging direction,

the control unit executes moving body detection, gesture detection, andtracking of the object for the first image, and

the control unit controls output of at least either the first image orthe second image in accordance with a result of any one of the movingbody detection, the gesture detection, and the tracking for the firstimage or an output change of the second image outputted from aninformation processing apparatus.

-   (25)

A control method of a controller, including:

controlling output of at least either a first image acquired by imagingin a first imaging direction or a second image different from the firstimage in accordance with at least a situation of an object associatedwith the first image.

REFERENCE SIGNS LIST

-   11 Camera-   11-1 Lecturer camera-   11-2 Auditor camera-   11-3 Board camera-   12 Camera controller-   12-1 Lecturer camera controller-   12-2 Auditor camera controller-   12-3 Board camera controller-   13 Controller-   21 Moving body detection unit-   22 Gesture detection unit-   23 Tracking unit-   24 Camera controller-   31 Center control unit-   32 Scenario generation unit-   33 Switcher-   211 Board extraction unit-   301 PC for slide

1. A controller, comprising: processing circuitry configured to: receivea detection result indicating a status of an object detectable in afield of view of one or more imaging devices when the field of view ofthe one or more imaging devices is being controlled according to a firststate of a control scenario, the control scenario including a pluralityof states for controlling the field of view of the one or more imagingdevices; determine whether a shift condition for triggering a shift ofstate specified in an event descriptor of the first state is detectedaccording to at least the status of the object, the event descriptorfurther specifying a second state among the plurality of states as ashift destination associated with the shift condition; and in responseto the shift condition specified in the event descriptor beingdetermined as detected, control the field of view of the one or moreimaging devices according to a behavior descriptor of the second state,the behavior descriptor specifying actions for controlling the field ofview of the one or more imaging devices.
 2. The controller according toclaim 1, wherein the processing circuitry is configured to control thefield of view of the one or more imaging devices by performing one ormore of mechanical panning, mechanical tilting, optical zooming, andelectronical zooming of the one or more imaging devices.
 3. Thecontroller according to claim 1, wherein the behavior descriptorspecifies the actions that include controlling an angle of view and animaging direction of the one or more imaging devices.
 4. The controlleraccording to claim 1, wherein the status of the object includes anaction of the object.
 5. The controller according to claim 4, whereinthe object is a person, and the action of the object includes at leastone of standing up, sitting down, raising hand, moving, or emittingsound.
 6. The controller according to claim 1, wherein the processingcircuitry is configured to receive a detection result of executing, bythe processing circuitry, one or more of moving body detection, gesturedetection, and tracking of the object according to one or more imagescaptured by the one or more imaging devices.
 7. The controller accordingto claim 1, wherein the processing circuitry is configured to controloutput of one or more images captured by the one or more imaging devicesaccording to a current state among the plurality of states.
 8. Thecontroller according to claim 1, wherein the processing circuitry isconfigured to: extract contents presented on another object from theother object detectable in another field of view of another imagingdevice, the contents being presented on the other object in response toan interaction between the object and the other object, and the otherfield of view being different from the field of view; and control one ormore of the field of view and the other field of view in accordance withat least one of the detected status or the extracted contents.
 9. Thecontroller according to claim 8, wherein the object corresponds to aperson included in a first image captured by one of the one or moreimaging devices, and the other object corresponds to a blackboard orwhiteboard included in a second image captured by the other imagingdevice.
 10. The controller according to claim 1, wherein the shiftcondition includes a condition based on a change of the status of theobject.
 11. The controller according to claim 1, wherein the controlscenario is expressed by a state shift chart or a timeline diagram. 12.The controller according to claim 1, wherein the processing circuitry isconfigured to add or delete a member state of the plurality of states inthe control scenario or change description contents of the plurality ofstates on a basis of an operation by a user.
 13. The controlleraccording to claim 12, wherein the processing circuitry is configured tocontrol display of a graphic user interface (GUI) that receives additionand deletion of the member state of the plurality of states in thecontrol scenario or a change of the description contents of theplurality of states.
 14. A controller, comprising: processing circuitryconfigured to: detect a status of a first object detectable in a firstfield of view of a first imaging device or a second object detectable ina second field of view of a second imaging device, the second field ofview being different from the first field of view; extract contentspresented on a third object from the third object detectable in a thirdfield of view of a third imaging device, the contents being presented onthe third object in response to an interaction between the first objectand the third object, and the third field of view being different fromthe first field of view and the second field of view; and control one ormore of the first field of view, second field of view, and the thirdfield of view in accordance with at least one of the detected status orthe extracted contents.
 15. The controller according to claim 14,wherein the processing circuitry is configured to control the one ormore of the first field of view, second field of view, and the thirdfield of view by performing one or more of mechanical panning,mechanical tilting, optical zooming, and electronical zooming ofcorresponding one or more of the first imaging device, the secondimaging device, and the third imaging device.
 16. The controlleraccording to claim 14, wherein the first object corresponds to a firstperson included in a first image captured by the first imaging device,the second object corresponds to a second person included in a secondimage captured by the second imaging device, and the status of the firstobject or the second object includes an action of the first person orthe second person, including at least one of standing up, sitting down,raising hand, moving, or emitting sound.
 17. The controller according toclaim 16, wherein the third object corresponds to a blackboard orwhiteboard included in a third image captured by the third imagingdevice, and the processing circuitry is configured to output aside-by-side display or a picture-in-picture display of two of the firstimage, the second image, and the third image in accordance with thedetected status.
 18. A control method of a controller, comprising:receiving a detection result indicating a status of an object detectablein a field of view of one or more imaging devices when the field of viewof the one or more imaging devices is being controlled according to afirst state of a control scenario, the control scenario including aplurality of states for controlling the field of view of the one or moreimaging devices; determining, by processing circuitry of the controller,whether a shift condition for triggering a shift of state specified inan event descriptor of the first state is detected according to at leastthe status of the object, the event descriptor further specifying asecond state among the plurality of states as a shift destinationassociated with the shift condition; and in response to the shiftcondition specified in the event descriptor being determined asdetected, controlling, by the processing circuitry of the controller,the field of view of the one or more imaging devices according to abehavior descriptor of the second state, the behavior descriptorspecifying actions for controlling the field of view of the one or moreimaging devices.
 19. The control method according to claim 18, whereinthe controlling the field of view of the one or more imaging devicescomprises performing one or more of mechanical panning, mechanicaltilting, optical zooming, and electronical zooming of the one or moreimaging devices.
 20. The control method according to claim 18, whereinthe behavior descriptor specifies the actions that include controllingan angle of view and an imaging direction of the one or more imagingdevices.